Network camera field of view calibration method and system using deep residual network

Abstract

The present application relates to the technical field of computer vision and deep learning inference constraints, and discloses a network camera field of view calibration method and system of a deep residual network, the method comprising: acquiring calibration image data stream and target pixel coordinates, and generating a standard input tensor containing field of view position encoding; extracting multi-scale features and forming a cross-scale geometric feature vector by radial partition attention; firstly predicting candidate intrinsic parameters, extrinsic parameters and distortion parameters, then generating a projection residual distribution from the projection of a space reference point by the candidate parameters, performing secondary prediction by using residual feedback weighted features, and outputting final field of view calibration parameters by intrinsic parameter structure, rotation orthogonality, distortion monotonicity and depth positive value constraints. The present application is used for reducing non-physical deviation of calibration parameters under strong distortion and complex illumination.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning inference constraint technology, specifically to a method and system for calibrating the field of view of a network camera using deep residual networks. Background Technology

[0002] After network cameras are installed on-site, it is necessary to establish a mapping relationship between real spatial coordinates and image pixel coordinates. This mapping relationship is typically determined by the camera's intrinsic parameters, extrinsic parameters, and lens distortion parameters. Intrinsic parameters reflect the influence of focal length, principal point position, pixel scale, and tilt on the imaging plane; extrinsic parameters reflect the rotation and translation of the camera's local coordinate system relative to the on-site spatial coordinate system; and distortion parameters reflect the degree of deviation of the real lens from the ideal pinhole imaging model. For applications such as road traffic, park security, industrial production lines, and warehouse inspection, calibration parameters directly affect target landing point calculation, target size estimation, multi-camera trajectory stitching, 3D reconstruction, and boundary violation detection.

[0003] Traditional camera calibration processes typically rely on checkerboard corner points, dot arrays, known straight lines, vanishing points, or on-site reference markers. However, calibration images from network cameras in the field are often affected by uneven illumination, long-distance shooting, video compression, partial occlusion of calibration objects, lens edge blurring, and image noise. Pixel-level errors at local corner points or center points will enter the homography matrix solution, nonlinear optimization, and distortion model fitting processes. Since subsequent optimization processes assume the input reference point is reliable, local errors will continue to propagate along the parameter solution chain, causing coupled offsets of intrinsic, extrinsic, and distortion parameters.

[0004] Deep learning-based camera calibration schemes reduce the need for manual corner detection and can directly regress focal length, principal point, pose, or distortion parameters from images. However, ordinary deep networks tend to weaken pixel position constraints after multiple downsampling and global pooling. Camera calibration is not a typical image classification task; principal point offset is directly related to pixel coordinates, radial distortion is directly related to the distance from the pixel to the center of the field of view, and extrinsic pose is directly related to the perspective distribution of the reference point in the entire image. If the deep network compresses the entire image into a single global feature vector, it will lose the geometric differences between the central field of view, transitional field of view, and edge field of view.

[0005] Direct regression calibration of parameters in deep networks still faces physical feasibility risks. Intrinsic parameter matrices may contain negative focal lengths, out-of-bounds principal points, or degenerate scale terms; rotation matrices may not satisfy orthogonality, and may even form reflection matrices after singular value correction; distortion parameters may cause radius mapping to fold within the effective field of view; extrinsic translation parameters may cause the visible space reference point to fall behind the camera. Adding projection loss only during training cannot guarantee that candidate parameter errors during inference will be fed back to the feature selection process. If edge occlusion or local distortion anomalies exist in the field image, parameters obtained from a single forward inference may still propagate local errors to the global calibration result. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for calibrating the field of view of a network camera using a deep residual network. The technical problem to be solved by this invention is that when a network camera is in an environment with complex lighting, strong compressed bitstream, field of view edge distortion, and skewed installation posture, the existing single-step depth regression calibration method is prone to weakening the pixel position constraints during feature compression, and the projection error of the candidate calibration parameters cannot be fed back to the feature selection process during the inference stage, resulting in regional deviations or non-physical combinations of the final output intrinsic parameters, extrinsic parameters, and distortion parameters.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] Methods for calibrating the field of view of network cameras using deep residual networks include:

[0009] The calibration image data stream output by the network camera is obtained, and the target pixel coordinates corresponding to the preset spatial reference point are obtained. The calibration image data stream is subjected to resolution scaling, channel mean alignment and field of view position encoding to generate a standard input tensor containing image intensity information and pixel position constraint information.

[0010] The standard input tensor is input into a multi-scale feature extraction network with residual connections to extract feature tensors of multiple scales. Scale-related attention weights are generated based on the central region descriptors, edge region descriptors, and global descriptors of each scale feature tensor, and then fused to obtain a cross-scale geometric feature vector.

[0011] The cross-scale geometric feature vector is input into the parameter prediction network to obtain candidate intrinsic parameters, candidate rotation parameters, candidate translation parameters, and candidate distortion parameters. Based on the candidate intrinsic parameters, candidate rotation parameters, candidate translation parameters, and candidate distortion parameters, a differentiable projection is performed on the preset spatial reference point to generate a projection residual distribution.

[0012] The feature tensors of the multiple scales are weighted by residual feedback according to the projection residual distribution, and the calibration parameters to be constrained are predicted again based on the feature tensors after residual feedback weighting.

[0013] The intrinsic parameter structure constraint, rotation orthogonal constraint, distortion monotonic constraint, and depth positive value constraint are applied to the calibration parameters to generate the final field-of-view calibration parameters of the network camera.

[0014] As a preferred embodiment of the present invention, the field of view position encoding includes: generating a normalized horizontal coordinate channel, a normalized vertical coordinate channel, and a radial distance channel for each pixel position in the calibrated image data stream; and concatenating the normalized horizontal coordinate channel, the normalized vertical coordinate channel, the radial distance channel, and the image intensity channel after aligning the channel mean to obtain the standard input tensor.

[0015] As a preferred embodiment of the present invention, the multi-scale feature extraction network includes multiple residual convolution stages arranged in series. Each residual convolution stage includes a main path convolutional layer and a residual connection path. The main path convolutional layer performs spatial feature extraction on the input feature tensor. The residual connection path passes the input feature tensor or the input feature tensor after dimension matching to the output of the corresponding residual convolution stage and adds it element-wise with the feature tensor output by the main path convolutional layer to form a feature tensor of the corresponding scale.

[0016] As a preferred technical solution of the present invention, generating the scale-related attention weights includes: dividing the feature tensors of each scale into a central field of view, a transitional field of view, and an edge field of view according to the pixel position constraint information; performing region pooling on the central field of view, the transitional field of view, and the edge field of view respectively to generate a radial partition descriptor; and inputting the radial partition descriptor and the global pooling descriptor into a parameter-shared attention mapping network to output the scale-related attention weights.

[0017] As a preferred technical solution of the present invention, obtaining the candidate intrinsic parameters, the candidate rotation parameters, the candidate translation parameters, and the candidate distortion parameters includes: inputting the cross-scale geometric feature vector into a shared hidden layer, outputting candidate focal length parameters, candidate principal point parameters, and candidate tilt parameters through an intrinsic parameter prediction branch, and filling the candidate focal length parameters, candidate principal point parameters, and candidate tilt parameters into preset matrix positions to construct a candidate intrinsic parameter matrix; outputting candidate rotation vectors and candidate translation vectors through an extrinsic parameter prediction branch; and outputting candidate radial distortion parameters and candidate tangential distortion parameters through a distortion prediction branch.

[0018] As a preferred embodiment of the present invention, generating the projection residual distribution includes: transforming the preset spatial reference point to the camera local coordinate system based on the candidate rotation parameters and the candidate translation parameters; performing normalized imaging processing on the transformed spatial reference point; performing distortion mapping on the normalized imaging processing result based on the candidate distortion parameters; converting the distortion mapping result into candidate pixel coordinates based on the candidate intrinsic parameters; and mapping the coordinate difference between the candidate pixel coordinates and the target pixel coordinates to the corresponding field of view region to obtain the projection residual distribution.

[0019] As a preferred embodiment of the present invention, residual feedback weighting of the feature tensors at multiple scales includes: resampling the projected residual distribution to the feature plane corresponding to each scale feature tensor according to the downsampling ratio of each scale feature tensor relative to the calibration image data stream; calculating the center residual intensity, transition residual intensity, edge residual intensity, and scale residual intensity in each feature plane; inputting the center residual intensity, the transition residual intensity, the edge residual intensity, and the scale residual intensity into the feedback weight generation network to obtain residual feedback weights, and using the residual feedback weights to jointly weight the feature tensors at multiple scales in terms of region dimension and channel dimension.

[0020] As a preferred technical solution of the present invention, the intrinsic parameter structure constraint includes: assembling an intrinsic parameter matrix to be constrained based on the focal length parameter to be constrained, the principal point parameter to be constrained, and the tilt parameter to be constrained in the calibration parameters to be constrained; covering the elements below the main diagonal of the intrinsic parameter matrix to be constrained with zero values; correcting the focal length element to positive values; correcting the scale element to non-zero values; and restricting the principal point parameter to be within the effective pixel range according to the input image size.

[0021] As a preferred embodiment of the present invention, the rotational orthogonal constraint includes: converting the rotation vector in the calibration parameters to be constrained into an initial rotation matrix, performing singular value decomposition on the initial rotation matrix, constructing a direction correction matrix based on the determinant sign of the product of the left and right singular matrices obtained from the decomposition, and reconstructing a rotation matrix belonging to a special orthogonal group using the left singular matrix, the direction correction matrix, and the right singular matrix.

[0022] As a preferred technical solution of the present invention, the distortion monotonic constraint includes: setting multiple radius sampling positions within the effective imaging radius range of the normalized image plane, calculating the distorted radius corresponding to each radius sampling position based on the distortion parameter in the calibration parameters to be constrained, detecting whether the distorted radius corresponding to adjacent radius sampling positions satisfies an increasing relationship, and performing shrinkage correction on the distortion parameter when radius folding is detected.

[0023] As a preferred technical solution of the present invention, the positive depth constraint includes: based on the rotation parameter and translation parameter in the calibration parameters to be constrained, transforming the preset spatial reference point to the camera local coordinate system, detecting whether the depth component of the transformed spatial reference point is greater than zero, and when there is a spatial reference point with a depth component not greater than zero, correcting the translation parameter according to the minimum depth offset, or outputting a status indicator of reacquiring and calibrating the image data stream.

[0024] As a preferred embodiment of the present invention, the parameter prediction network is obtained through training. The training process includes: acquiring a set of historical calibration images containing three-dimensional spatial reference points with known spatial coordinates; performing geometric enhancement on the historical calibration image set in sync with the real calibration parameter labels; using the enhanced historical calibration image set for forward inference to obtain a set of predicted calibration parameters; calculating parameter regression loss, projection consistency loss, distortion monotonicity penalty loss, depth positive value penalty loss, and feedback effectiveness loss based on the set of predicted calibration parameters and the real calibration parameter labels; fusing the loss terms according to a dynamic weight strategy that changes with the training rounds to form a composite loss; and using the composite loss to update the network parameters of the parameter prediction network and the multi-scale feature extraction network.

[0025] In addition, this invention also discloses a network camera field-of-view calibration system for deep residual networks, used for the network camera field-of-view calibration method for deep residual networks as described above, comprising:

[0026] The preprocessing module is used to acquire the calibration image data stream output by the network camera, and to acquire the target pixel coordinates corresponding to the preset spatial reference point. The calibration image data stream is then subjected to resolution scaling, channel mean alignment, and field of view position encoding to generate a standard input tensor containing image intensity information and pixel position constraint information.

[0027] The multi-scale feature extraction module is used to input the standard input tensor into a multi-scale feature extraction network with residual connections to extract feature tensors at multiple scales.

[0028] The radial attention fusion module is used to generate scale-related attention weights based on the central region descriptor, edge region descriptor and global descriptor of the feature tensor at each scale, and fuse them to obtain a cross-scale geometric feature vector;

[0029] The candidate parameter prediction module is used to input the cross-scale geometric feature vector into the parameter prediction network to obtain candidate intrinsic parameters, candidate rotation parameters, candidate translation parameters, and candidate distortion parameters.

[0030] The projection residual feedback function module is used to perform differentiable projection on the preset spatial reference point based on the candidate intrinsic parameters, the candidate rotation parameters, the candidate translation parameters and the candidate distortion parameters, generate a projection residual distribution, and perform residual feedback weighting on the feature tensors of the multiple scales according to the projection residual distribution;

[0031] The quadratic parameter prediction module is used to predict the calibration parameters to be constrained again based on the feature tensor after residual feedback weighting.

[0032] The physical constraint output function module is used to perform intrinsic parameter structural constraints, rotational orthogonal constraints, distortion monotonic constraints, and depth positive constraints on the calibration parameters to be constrained, so as to generate the final field-of-view calibration parameters of the network camera.

[0033] Meanwhile, the present invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the network camera field-of-view calibration method for deep residual networks as described above.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] This invention's field-of-view position encoding incorporates normalized horizontal coordinates, normalized vertical coordinates, and radial distance into a standard input tensor, enabling the multi-scale feature extraction network to retain pixel position constraints while extracting texture, edge, and calibration structure features. Radial partitioning attention processes the central, transitional, and edge field-of-view regions separately, allowing the influence of edge distortion regions on distortion parameters and principal point parameters to be expressed independently. Candidate calibration parameters are projected onto a preset spatial reference point to form a projection residual distribution, which is fed back to the multi-scale feature tensor, ensuring that the geometric errors in the first prediction effectively control the second feature weighting and parameter prediction. Intrinsic structural constraints, rotational orthogonality constraints, distortion monotonicity constraints, and depth positivity constraints collectively limit the final field-of-view calibration parameters to the physically feasible region of the camera imaging model, reducing the risk of non-physical intrinsic matrix errors, reflection rotation matrices, distortion radius folding, and reference points falling behind the camera. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is a flowchart of the method described in this invention.

[0038] Figure 2 This is a system architecture environment diagram of the present invention.

[0039] Figure 3 This is a flowchart of the scale-related attention weight generation sub-process of the present invention.

[0040] Figure 4 This is a flowchart of the candidate calibration parameter acquisition sub-process of the present invention.

[0041] Figure 5 This is a flowchart of the projection residual distribution generation sub-process of the present invention.

[0042] Figure 6 This is a flowchart of the residual feedback weighting sub-process of the present invention.

[0043] Figure 7 This is a flowchart of the parameter physical constraint processing sub-process of the present invention.

[0044] Figure 8 This is a flowchart of the parameter prediction network training process of the present invention. Detailed Implementation

[0045] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0046] The following is in conjunction with the appendix Figures 1-8 The embodiments of the present invention will be described in detail below.

[0047] Example 1: This example provides a method for calibrating the field of view of a network camera using a deep residual network. The method can be deployed in an edge computing box, a network camera's built-in processor, a video management server, or an industrial vision computing device. The network camera can be a bullet network camera, a dome network camera, a wide-angle network camera, a fisheye network camera, or a fixed-mount camera with a zoom lens.

[0048] The inputs in this embodiment include the calibration image data stream output by the network camera, preset spatial reference points, and target pixel coordinates. The calibration image data stream can come from a planar calibration board, a three-dimensional calibration frame, known ground markings, known reference points on a wall, a combination of road lane lines and marker points, or it can be composed of reference objects with known three-dimensional coordinates on site. The preset spatial reference points are a set of three-dimensional spatial points corresponding to the calibration image data stream.

[0049] In practice, during the initial calibration scenario, the target pixel coordinates are obtained through manual confirmation, high-precision offline corner point extraction process, structured light measurement results, or laser ranging-assisted results.

[0050] In recalibration or online verification scenarios, the target pixel coordinates can be obtained by projecting historically reliable calibration parameters onto a spatial reference point, and consistency verification can be performed using local matching results in the current image. This distinction avoids the initial calibration process relying on the final field-of-view calibration parameters that have not yet been generated.

[0051] The specific steps are as follows:

[0052] Step S101: Obtain the calibration image data stream output by the network camera, and obtain the target pixel coordinates corresponding to the preset spatial reference point.

[0053] The preprocessing module reads image frames from the network camera's video stream interface. The video stream interface can be a real-time streaming protocol interface, a hypertext transfer interface, an open network video interface, an industrial camera acquisition interface, or a local cache file interface. The preprocessing module decodes the read compressed image frames to obtain the original image intensity matrix. The original image intensity matrix includes red, green, and blue channels, and may also include grayscale channels or a luminance channel obtained after luminance-chrominance separation.

[0054] The preprocessing module synchronously reads the preset spatial reference point set and the target pixel coordinate set. Each spatial reference point in the preset spatial reference point set has on-site spatial coordinates. Each target pixel coordinate in the target pixel coordinate set corresponds to a spatial reference point. If the target pixel coordinates come from the calibration board image, the target pixel coordinates can be manually confirmed or generated by a high-precision corner point extraction process; if the target pixel coordinates come from the road or park's on-site structure, the target pixel coordinates can be selected by maintenance personnel in the image and bound to the on-site measurement coordinates; if the target pixel coordinates come from historical reliable calibration parameters in a recalibration or online verification scenario, the target pixel coordinates need to be consistent with the local matching results in the current image before participating in the generation of the projection residual distribution.

[0055] For continuous video streams, the preprocessing module does not need to process every single frame. Instead, it selects calibration image frames from the calibration image data stream that meet the acquisition criteria based on inter-frame sharpness, motion blur, and reference point visibility. These criteria include average image brightness within a valid range, a reference area unobstructed ratio meeting set requirements, Laplacian gradient response meeting sharpness requirements, and video coding block artifacts not exceeding set limits. Image frames that do not meet these criteria are recorded as low-confidence input frames and do not enter the parameter prediction network.

[0056] Step S102: Perform resolution scaling and channel mean alignment on the calibration image data stream;

[0057] The preprocessing module scales the calibrated image frames to a uniform input size. Scaling methods can include bilinear interpolation, region resampling, or aspect ratio-preserving padding. For wide-angle network cameras, direct stretching would alter edge distortion distribution; therefore, the preprocessing module prioritizes aspect ratio-preserving padding and records the effective image region mask. This effective image region mask is used to exclude padded regions during subsequent region pooling, preventing padded values ​​from affecting scale-related attention weights.

[0058] Channel mean alignment is used to reduce the impact of different illumination color temperatures on feature extraction. The preprocessing module calculates the mean and variance of each image intensity channel and performs mean subtraction and scale normalization on each channel. In some implementations, the preprocessing module can use the channel mean and channel scale factor obtained from the training dataset; in other implementations, it can use the channel statistics of the current calibration image frame and smoothly fuse them with historical statistics. The smoothed statistics are written into the input metadata for recording ambient light changes during the confidence assessment process.

[0059] Step S103: Encode the field of view position of the calibration image data stream to generate a standard input tensor;

[0060] Camera field-of-view calibration depends on pixel positions. Principal point offset corresponds to a change in the center position in the image coordinate system. The distortion amplitude is usually related to the radial distance from the pixel to the principal point or the image center. Mounting posture deviations will cause different perspective distortions in different image regions. Therefore, this embodiment constructs field-of-view position encoding outside the image intensity channel.

[0061] The preprocessing module generates a normalized horizontal coordinate channel for each pixel location. This channel records the relative position of the current pixel with respect to the horizontal center of the image. The preprocessing module also generates a normalized vertical coordinate channel for each pixel location. This channel records the relative position of the current pixel with respect to the vertical center of the image. Based on the normalized horizontal and vertical coordinates, the preprocessing module generates a radial distance channel. This channel records the degree to which the current pixel is located in the central field of view, transitional field of view, or edge field of view.

[0062] In some implementations, the preprocessing module also generates radial partitioning masks. These radial partitioning masks include a central field-of-view mask, a transitional field-of-view mask, and an edge field-of-view mask. The central field-of-view corresponds to a set of pixels with smaller radial distances, the edge field-of-view corresponds to a set of pixels with larger radial distances, and the transitional field-of-view is located between the central and edge field-of-view regions. For fisheye network cameras, the preprocessing module can also generate an effective circular domain mask based on the effective circular field-of-view boundary to avoid black border regions participating in feature statistics.

[0063] The preprocessing module concatenates the image intensity channel, normalized horizontal coordinate channel, normalized vertical coordinate channel, and radial distance channel after channel mean alignment to generate a standard input tensor. This standard input tensor contains image intensity information such as texture, edges, and morphology of the calibrated object, as well as pixel position constraints related to principal point offset and radial distortion. The standard input tensor is then fed into the multi-scale feature extraction module.

[0064] Step S104: Extract feature tensors of multiple scales through a multi-scale feature extraction network with residual connections;

[0065] The multi-scale feature extraction module executes a multi-scale feature extraction network. This network comprises multiple cascaded residual convolutional stages. The first residual convolutional stage receives a standard input tensor and extracts local edges, object contours, ground lines, wall reference textures, and field-of-view edge curvature features. Subsequent residual convolutional stages gradually expand the receptive field, extracting high-level geometric features related to perspective direction, relative layout of reference points, radial distortion trends, and installation posture.

[0066] Each residual convolutional stage consists of a main path convolutional layer and residual connection paths. The main path convolutional layer performs convolution, normalization, and nonlinear transformations on the input feature tensor to obtain the main path output feature tensor. The residual connection paths directly pass the input feature tensor to the output; when the spatial dimensions or number of channels of the input feature tensor and the main path output feature tensor are inconsistent, the residual connection paths perform dimension matching. The multi-scale feature extraction network adds the main path output feature tensor and the residual connection path output feature tensor element-wise to obtain the output feature tensor of the corresponding residual convolutional stage.

[0067] The role of residual connection paths is to preserve shallow location and edge information, allowing subtle geometric offsets in the calibration image to continue to propagate to deeper layers. For heavily compressed bitstreams, shallow features may contain block boundary noise; for wide-angle lenses, shallow edge curvature information is crucial for distortion estimation. This embodiment preserves shallow features through residual connection paths, and then uses subsequent radial partitioning attention to determine which regions and channels should be enhanced, rather than discarding shallow information directly at the front end.

[0068] The multi-scale feature extraction network outputs feature tensors at multiple scales at different downsampling depths. Shallow-scale feature tensors retain more spatial details and are used to describe the edges of reference points, local line segments, and the geometry near the principal point; deeper-scale feature tensors contain a larger receptive field and are used to describe the overall perspective relationship of the calibration object, the camera's mounting posture, and the field of view curvature. All feature tensors at multiple scales are then passed to the radial attention fusion module.

[0069] Step S105: Generate radial partition descriptors and global descriptors.

[0070] The radial attention fusion module receives feature tensors at multiple scales and field-of-view position codes generated by the preprocessing module. Since the feature tensors are downsampled, the radial attention fusion module synchronously scales the central field-of-view mask, transitional field-of-view mask, and edge field-of-view mask to the spatial dimensions corresponding to the feature tensors at each scale. The scaled region masks are then aligned with the feature tensors at their respective scales.

[0071] For a feature tensor of any scale, the radial attention fusion module performs region pooling in the central field of view, transitional field of view, and edge field of view. Region pooling can include region average pooling, region max pooling, and region second-moment statistics. Region average pooling describes the overall response of the feature channels within the corresponding region; region max pooling describes the local high-response reference structure; and region second-moment statistics describe the dispersion of the feature response within the region. The results of these three types of region pooling together form the radial partition descriptor.

[0072] The radial attention fusion module also performs global pooling on the complete feature tensor to generate a global descriptor. The global descriptor is used to represent the global perspective relationships in the entire image. The radial partition descriptor and the global descriptor are concatenated and input parameters are shared by the attention mapping network. The attention mapping network outputs scale-related attention weights. Scale-related attention weights include channel dimension weights, radial region dimension weights, and scale dimension weights.

[0073] Instead of simply performing global average pooling to remove spatial location information, this embodiment compresses spatial locations into radial partition descriptors and a global descriptor. This process reduces the tensor dimension while preserving the different contributions of the central field of view, transitional field of view, and edge field of view to the calibration parameters.

[0074] Step S106: The cross-scale geometric feature vector is obtained by fusion.

[0075] The radial attention fusion module uses scale-related attention weights to weight the feature tensors at each scale. For feature channels that are sensitive to residuals in the edge field of view, scale-related attention weights can improve the response in the edge region; for irrelevant channels that are greatly affected by compressed noise, scale-related attention weights can reduce the channel response. The weighted feature tensors at each scale are then spatially compressed, channel-wise processed, and scale-wise concatenated to generate cross-scale geometric feature vectors.

[0076] The cross-scale geometric feature vector is not simply an image classification feature, but a combined representation that includes image intensity, pixel location, radial partitioning, low-level edges, high-level perspective relationships, and scale correlation weights. The cross-scale geometric feature vector is then sent to the candidate parameter prediction module.

[0077] Step S107: Predict candidate calibration parameters based on cross-scale geometric feature vectors and construct candidate intrinsic parameter matrices.

[0078] The candidate parameter prediction module includes a shared hidden layer, an intrinsic parameter prediction branch, an extrinsic parameter prediction branch, and a distortion prediction branch. The shared hidden layer receives cross-scale geometric feature vectors and outputs a shared geometric latent vector. The shared geometric latent vector is distributed to the three prediction branches.

[0079] The intrinsic parameter prediction branch outputs candidate focal length parameters, candidate principal point parameters, and candidate tilt parameters. Candidate focal length parameters include candidate horizontal focal length parameters and candidate vertical focal length parameters. Candidate principal point parameters include the horizontal and vertical coordinates of the candidate principal point. The candidate tilt parameter corresponds to the candidate tilt term when the pixel coordinate axes are not orthogonal. The candidate parameter prediction module fills the candidate horizontal focal length parameters into the first row and first column of the candidate intrinsic parameter matrix according to the preset matrix positions, fills the candidate vertical focal length parameters into the second row and second column, fills the candidate tilt parameters into the first row and second column, fills the candidate principal point horizontal coordinates into the first row and third column, fills the candidate principal point vertical coordinates into the second row and third column, and sets the third row and third column of the candidate intrinsic parameter matrix to a unit scale value. The position below the main diagonal of the candidate intrinsic parameter matrix is ​​set to zero in subsequent intrinsic parameter structure constraints.

[0080] The extrinsic parameter prediction branch outputs candidate rotation vectors and candidate translation vectors. The candidate rotation vectors can be in Rodriguez parameterized form, or represented using quaternions or six-dimensional rotation. In this embodiment, the candidate rotation vectors are first converted into candidate rotation matrices, which are then subjected to rotational orthogonality constraints. The candidate translation vectors represent the translation relationship between the network camera's local coordinate system and the field coordinate system.

[0081] The distortion prediction branch outputs candidate radial distortion parameters and candidate tangential distortion parameters. The candidate radial distortion parameters describe lens distortion that varies with the radius, while the candidate tangential distortion parameters describe asymmetric distortion introduced by the lens-image plane assembly misalignment. In some implementations, the distortion prediction branch may also output a field-of-view ray lookup table, from which the distortion parameters are then fitted.

[0082] Step S108: Perform differentiable projection on the preset spatial reference point based on the candidate calibration parameters.

[0083] The projection residual feedback module receives candidate intrinsic parameter matrices, candidate rotation parameters, candidate translation parameters, candidate distortion parameters, preset spatial reference points, and target pixel coordinates. The preset spatial reference points can be the 3D coordinates of calibration board corner points, the 3D coordinates of the circular center array, the 3D coordinates of ground marking sampling points, the 3D coordinates of wall reference points, or the 3D coordinates of on-site structural reference points.

[0084] The projection residual feedback module first transforms the preset spatial reference point to the camera's local coordinate system based on candidate rotation and translation parameters, and then performs normalized imaging processing according to the depth component in the camera's local coordinate system. The processing procedure is as follows:

[0085]

[0086] in, For the first The horizontal coordinates of each preset spatial reference point in the on-site spatial coordinate system, in meters; For the first The vertical coordinates of each preset spatial reference point in the on-site spatial coordinate system, in meters; For the first The depth coordinates of each preset spatial reference point in the on-site spatial coordinate system, in meters; For the first The horizontal coordinates of each preset spatial reference point after being transformed to the camera's local coordinate system are in meters. For the first The vertical coordinates of each preset spatial reference point after being transformed to the camera's local coordinate system are in meters. For the first The depth coordinates of a preset spatial reference point after being transformed to the camera's local coordinate system are in meters. Let be a candidate rotation matrix, dimensionless, belonging to an orthogonal matrix; These are candidate translation vectors, in meters (m). For the first Normalized horizontal coordinates of a preset spatial reference point, dimensionless; For the first Normalized vertical coordinates of a preset spatial reference point, dimensionless; This is the preset spatial reference point number. , This represents the total number of preset spatial reference points.

[0087] Differentiable projection means that coordinate transformations, normalized imaging, distortion mapping, and pixel projection are all composed of differentiable operators. During the model training phase, differentiable projection is used for backpropagation of errors; during the model inference phase, differentiable projection is used as a deterministic geometric projection calculation process and does not require gradient updates.

[0088] The projection residual feedback module performs distortion mapping on the normalized horizontal and normalized vertical coordinates based on candidate radial and tangential distortion parameters. Radial distortion mapping performs multiplicative scaling on the normalized coordinates according to the radius from the center of the field of view, while tangential distortion mapping performs additive offset based on the coupling relationship between the normalized horizontal and normalized vertical coordinates. The distortion mapping results in distorted normalized horizontal and vertical coordinates.

[0089] The projection residual feedback module converts the distorted normalized horizontal and vertical coordinates into candidate pixel coordinates based on the candidate intrinsic parameter matrix. During the conversion, the candidate horizontal focal length parameter is applied to the distorted normalized horizontal coordinates, the candidate vertical focal length parameter is applied to the distorted normalized vertical coordinates, and the candidate tilt parameter is applied to the distorted normalized vertical coordinates and superimposed on the candidate pixel horizontal coordinates. The candidate principal point horizontal and vertical coordinates are added to the candidate pixel coordinates as pixel coordinate offsets.

[0090] A projection residual is formed between the candidate pixel coordinates and the target pixel coordinates. The projection residual includes horizontal residual, vertical residual, and residual magnitude. The projection residual feedback module maps the projection residual to the central field of view, transitional field of view, or edge field of view, based on the image region where the candidate pixel coordinates are located, forming a projection residual distribution. The projection residual distribution is not a single scalar, but rather a residual intensity map based on a two-dimensional image region. The residual intensity map records the degree of inconsistency of the candidate calibration parameters in different field of view regions.

[0091] Step S109: Map the projection residual distribution to a multi-scale feature tensor and generate the scale residual intensity.

[0092] Since the projection residual distribution lies in the original image coordinate domain, while the multi-scale feature tensors lie in the feature coordinate domains at different downsampling ratios, the projection residual feedback module performs cross-scale resampling. For any scale feature tensor, the projection residual feedback module converts the candidate pixel coordinates to their corresponding coordinates in the feature plane at that scale, based on the downsampling ratio of the feature tensor relative to the input image. The converted residual points generate a scale residual map in the feature plane using a neighborhood diffusion method. The neighborhood diffusion method can employ bilinear interpolation, nearest neighbor assignment, or Gaussian local diffusion.

[0093] When the candidate pixel coordinates fall at a non-integer position on the corresponding scale feature plane, the projection residual feedback module distributes the projection residuals to the four adjacent feature grid positions using bilinear interpolation weights. Through this process, the projection residuals in the two-dimensional pixel domain can be aligned with the spatial positions of feature tensors at different scales.

[0094] After obtaining the scale residual map, the projection residual feedback module calculates the center residual intensity, transition residual intensity, edge residual intensity, and scale residual intensity by combining the central field-of-view mask, transition field-of-view mask, edge field-of-view mask, and effective area mask. The center residual intensity is determined by the statistical values ​​of the scale residual map within the central field-of-view region; the transition residual intensity is determined by the statistical values ​​of the scale residual map within the transition field-of-view region; the edge residual intensity is determined by the statistical values ​​of the scale residual map within the edge field-of-view region; and the scale residual intensity is determined by the statistical values ​​of the scale residual map within the complete effective area. The statistical values ​​can be the average, weighted average, maximum value, or quantile value.

[0095] The aforementioned residual intensity allows the projection error in the pixel domain to flow back to feature tensors at different scales. Through this mapping process, the projection residual no longer remains on the two-dimensional image plane, but becomes a feedback signal that affects the multi-scale feature weighting.

[0096] Step S110: Generate residual feedback weights based on the projection residual distribution.

[0097] The projection residual distribution reflects the geometric errors of the first candidate parameters in different fields of view. If the residual is small in the central field of view but large in the peripheral field of view, it indicates that the candidate distortion parameters or principal point parameters may not be able to explain the curvature of the edge regions. If the vertical residuals are concentrated in the upper region of the image, it indicates that there may be deviations in the candidate pitch angles or the vertical coordinates of the candidate principal points. If the residuals of reference points corresponding to multiple scales are concentrated in the shallow texture region, it indicates that the shallow edge features are not being fully utilized.

[0098] The projection residual feedback module inputs the center residual intensity, transition residual intensity, edge residual intensity, and scale residual intensity into the feedback weight generation network. The feedback weight generation network outputs residual feedback weights. These residual feedback weights include region feedback weights, channel feedback weights, and scale feedback weights. Region feedback weights are used to increase the feature participation in regions with concentrated residuals; channel feedback weights are used to select feature channels related to geometric errors; and scale feedback weights are used to adjust the proportion of shallow spatial details and deep global perspective relationships in the secondary prediction.

[0099] Feedback weight generation networks can employ a two-layer fully connected mapping structure, a lightweight multilayer perceptron structure, or a gated mapping structure. The input to the feedback weight generation network is a combination vector of residual intensity, and the output, after normalization, forms feedback weights of different scales, channels, and regions. The training objective of the feedback weight generation network is not simply to amplify the residual values, but rather to learn the correspondence between the projection residual distribution and the effective feature correction direction.

[0100] Step S111: Perform residual feedback weighting on the multi-scale feature tensor.

[0101] The projection residual feedback module transmits residual feedback weights to the radial attention fusion module and the quadratic parameter prediction module. The radial attention fusion module applies region feedback weights to the central, transitional, and edge field-of-view regions corresponding to each scale feature tensor; applies channel feedback weights to the channel dimensions of each scale feature tensor; and applies scale feedback weights to the fusion ratio between feature tensors of different scales.

[0102] Residual feedback weighting does not simply amplify pixels in the residual region, but rather gives higher feature representation weights to fields of view that have not been adequately explained by the candidate parameters in the secondary prediction. If the projection error of the first candidate parameters to the edge region is large, the distortion-related features of the edge field of view will be enhanced; if the error of the first candidate parameters near the principal point in the central region is large, the related channels of the central field of view and the principal point will be enhanced; if the first candidate parameters have a consistent deviation in the overall perspective direction, the weight of deep-scale features will be enhanced.

[0103] Step S112: Based on the residual feedback weighted feature tensor, predict the calibration parameters to be constrained again.

[0104] The secondary parameter prediction module receives the feature tensor weighted by residual feedback. This module can share some network parameters with the candidate parameter prediction module, or it can use a separate secondary prediction head. In this embodiment, the secondary parameter prediction module receives at least two types of input: the cross-scale geometric feature vector weighted by residual feedback, and the candidate calibration parameters obtained from the first prediction. The secondary parameter prediction module can utilize both the new feature weights and the reference positions of the first candidate parameters.

[0105] The quadratic parameter prediction module outputs the calibration parameters to be constrained. These parameters include the focal length, principal point, tilt, rotation, translation, and distortion parameters. While the regional errors have been reduced by the projection residual feedback, non-physical issues such as matrix structure, rotational orthogonality, distortion monotonicity, or reference point depth may still exist. Therefore, the calibration parameters also need to be input into the physical constraint output module.

[0106] Step S113: Apply intrinsic parameter structural constraints to the calibration parameters to be constrained. The physical constraint output module constructs the intrinsic parameter matrix to be constrained based on the focal length parameter, principal point parameter, and tilt parameter to be constrained. The intrinsic parameter matrix to be constrained is used to convert the distortion-corrected normalized coordinates into pixel coordinates. The construction method of the intrinsic parameter matrix to be constrained is the same as that of the candidate intrinsic parameter matrix, except that the candidate parameters are replaced with the parameters to be constrained.

[0107] The physical constraint output module covers the elements below the main diagonal of the intrinsic parameter matrix to be constrained with zero values, ensuring the matrix satisfies an upper triangular structure. It also corrects the horizontal and vertical focal length elements to positive values, ensuring the focal length has a valid length in the physical direction. Furthermore, it corrects the scale elements to non-zero values ​​to prevent pixel coordinate mapping degradation. Finally, it limits the principal point parameters to within the effective pixel range based on the input image size.

[0108] Intrinsic parameter structure constraints are not expressed as positive definite matrix constraints, but directly correspond to the upper triangular structure, positive focal length, non-zero scale, and effective principal point range in real camera perspective imaging. Intrinsic parameter structure constraints can prevent neural networks from outputting negative focal lengths, out-of-bounds principal points, or degenerate intrinsic parameter matrices when there are insufficient training samples or abnormal field images.

[0109] Step S114: Apply rotational orthogonal constraints to the parameters to be constrained and calibrated. The physical constraint output module converts the rotation parameters to be constrained into an initial rotation matrix. Due to errors in neural network output and floating-point calculations, the initial rotation matrix may not satisfy strict orthogonality. Without correction, spatial points may exhibit non-rigid deformation after transformation by the initial rotation matrix, thus disrupting three-dimensional distance and angular relationships.

[0110] The rotation orthogonality constraint is reconstructed using singular value decomposition and a direction correction matrix, as detailed below:

[0111]

[0112] in, The initial rotation matrix is ​​obtained from the transformation of the rotation parameters to be constrained; it is dimensionless and belongs to... ; The left singular matrix obtained from the singular value decomposition of the initial rotation matrix is ​​dimensionless and belongs to... an orthogonal matrix; The singular value matrix obtained from the singular value decomposition of the initial rotation matrix is ​​dimensionless and belongs to... a diagonal matrix; The right singular matrix obtained from the singular value decomposition of the initial rotation matrix is ​​dimensionless and belongs to... an orthogonal matrix; It is the transpose of the right singular matrix; The direction correction matrix is ​​dimensionless and belongs to... a diagonal matrix; Represented by numerical value Numerical values and determinant A diagonal matrix constructed using the main diagonal elements; For matrix The determinant of is dimensionless; Let be the rotation matrix after rotational orthogonal constraints, which is dimensionless and belongs to . Special orthogonal groups .

[0113] The orientation correction matrix is ​​used to prevent the reconstruction result from becoming a reflection matrix with a negative determinant. The rotation matrix after rotation orthogonality constraint satisfies the requirements of rigid body rotation: row vectors are orthogonal, column vectors are orthogonal, and the determinant is positive.

[0114] Step S115: Perform distortion monotonic constraints on the parameters to be calibrated: The physical constraint output module sets multiple radius sampling positions within the effective imaging radius range of the normalized image plane. The effective imaging radius range refers to the effective field of view radius range used for calibration and projection measurement in the network camera output image, excluding fisheye black border areas, invalid masking areas, and edge areas marked as unusable for geometric measurement in the device configuration file.

[0115] The radius sampling position extends from the center of the field of view to the boundary of the effective imaging radius. The physical constraint output function module calculates the distorted radius corresponding to each radius sampling position based on the distortion parameters to be constrained. Within the effective imaging radius, a larger normalized radius should not be mapped to a position closer to the center than the adjacent smaller radius; otherwise, radius folding will occur. Radius folding can cause multiple spatial rays to be incorrectly mapped to the same or reversed pixel regions, making edge target localization unusable.

[0116] The distortion monotonicity detection is based on whether the distorted radius at adjacent radius sampling positions remains increasing. If the distorted radius at a more distant radius sampling position is not greater than the distorted radius at an adjacent, closer radius sampling position, the physical constraint output module determines that radius folding has occurred. If radius folding is detected, the physical constraint output module performs shrinkage correction on the distortion parameters. Shrinkage correction can be achieved by reducing the amplitude of higher-order radial distortion parameters, lowering the order of the distortion model, or pulling the distortion parameters back to the parameter region that most recently satisfies the monotonicity condition. The distortion monotonicity constraint ensures that the final distortion parameters will not form a non-physical radius inversion mapping.

[0117] Step S116: Apply positive depth constraints to the calibration parameters to be constrained: The physical constraint output module transforms the preset spatial reference points to the camera's local coordinate system based on the rotation and translation parameters to be constrained. If the depth component of some preset spatial reference points is not greater than zero, it indicates that the corresponding spatial reference points fall behind the camera's imaging plane or in a degraded imaging position. For normal calibration images, reference points that can be captured by the camera should be located within the effective field of view in front of the camera.

[0118] In this embodiment, the translation vector to be constrained is the translation vector used when transforming from the field space coordinate system to the camera's local coordinate system. The third component of the translation vector to be constrained corresponds to the depth direction in the camera's local coordinate system. Let the depth component after the transformation of the rotation and translation parameters to be constrained be... The minimum depth offset and the translation vector after depth direction correction are determined as follows:

[0119]

[0120] in, This is the minimum depth offset, in meters (m). For the first The depth components of a preset spatial reference point after being transformed into the camera's local coordinate system under the constraint calibration parameters, in meters; The preset minimum effective depth threshold is in meters and is a positive value. This represents the total number of preset spatial reference points; This is the preset spatial reference point number. ; This is a safety margin for depth to avoid the depth component from approaching zero, in meters, and taken as a small positive value; This is the translation vector after applying the positive depth constraint, in meters. Let be the translation vector to be constrained, in meters (m). This indicates that the larger value between zero and insufficient depth is taken, in meters. This represents the minimum value among all preset spatial reference point depth components, in meters (m).

[0121] If the minimum depth offset is greater than the depth safety margin, the physical constraint output module corrects the translation parameters to be constrained along the depth direction of the camera's local coordinate system. If the corrected projection residual still does not meet the confidence condition, the physical constraint output module outputs a status flag for reacquiring and calibrating the image data stream, instead of outputting incorrect final field-of-view calibration parameters.

[0122] Step S117: Generate the final field-of-view calibration parameters and calibration confidence level.

[0123] The physical constraint output module combines the intrinsic parameter matrix (constrained by intrinsic parameter structure constraints), the rotation matrix (constrained by rotation orthogonality constraints), the translation vector (constrained by positive depth constraints), and the distortion parameters (constrained by distortion monotonicity constraints) into the final field-of-view calibration parameters. These final field-of-view calibration parameters can be output as a matrix file, structured data packet, network interface response, device configuration file, or application programming interface (API) return value.

[0124] The physical constraint output module also generates calibration confidence based on the projection residual distribution. Calibration confidence can be determined by the mean projection residual, maximum projection residual, edge residual intensity, residual region concentration, intrinsic parameter correction magnitude, rotation correction magnitude, distortion shrinkage magnitude, and depth correction magnitude. If the calibration confidence is lower than the confidence threshold, the system outputs the final field-of-view calibration parameters along with a low-confidence flag, or only outputs a status flag indicating that the calibration image data stream has been reacquired. When the downstream target localization module receives a low-confidence flag, it does not use the corresponding calibration results to update the production environment parameters.

[0125] This embodiment, through the aforementioned process, eliminates the reliance on a single network output for network camera field-of-view calibration. The first set of candidate parameters reveals the geometric residual, which is then used to adjust the characteristics. A second prediction is then passed through the physical feasible region constraint for output. All outputs from previous stages are used in subsequent stages, forming a continuous closed loop.

[0126] Example 2: This example is a further optimization based on Example 1. In this example, the training process of the multi-scale feature extraction network, attention mapping network, candidate parameter prediction module, feedback weight generation network and secondary parameter prediction module is described.

[0127] Step S201: Construct a historical calibration image set. The training device acquires the historical calibration image set. This set includes calibration images captured by different types of network cameras. The acquisition conditions cover different focal lengths, principal point positions, distortion intensities, installation heights, pitch angles, roll angles, illumination levels, bitrate compression ratios, and occlusion ratios. For each historical calibration image, the training device records the corresponding true calibration parameter labels. These labels include true intrinsic parameters, true rotation parameters, true translation parameters, and true distortion parameters.

[0128] True calibration parameter labels can be obtained through a high-precision offline calibration process. This offline calibration process uses multi-pose calibration boards, 3D calibration frames, laser ranging-assisted coordinate systems, or precision measuring equipment to confirm the coordinates of spatial reference points. To avoid excessive noise in the training labels, the training equipment performs reprojection error filtering on the offline calibration results. Images with reprojection errors exceeding the labeling quality threshold are either not included in the training images or are only used as outlier samples for confidence-gated training.

[0129] In this embodiment's training configuration, the total number of samples is 16,200, including 12,000 training images, 2,400 verification images, and 1,800 independent test images. The training images include 4,800 images from ordinary field-of-view network cameras, 4,200 images from wide-angle network cameras, and 3,000 images from fisheye approximation network cameras. The independent test images include 600 ordinary field-of-view images, 700 wide-angle field-of-view images, and 500 fisheye approximation field-of-view images. Each image corresponds to 36 to 96 preset spatial reference points. The training images, verification images, and independent test images are isolated according to the network camera number and installation pose; consecutive images from the same network camera at the same installation pose are not simultaneously included in the training, verification, and independent test image sets.

[0130] Step S202: Perform synchronous geometric augmentation on the historical calibration image set: The training device performs geometric augmentation on the historical calibration images. Geometric augmentation includes horizontal mirroring, vertical cropping, proportional scaling, boundary padding, slight rotation, brightness variation, contrast variation, noise compression simulation, and local occlusion simulation. Geometric augmentation cannot only change the image pixels; it must also synchronously transform the true calibration parameter labels.

[0131] When performing horizontal mirroring enhancement, the training device corrects the principal point's horizontal coordinates based on the original image width, performs a sign transformation on the tilt term in the intrinsic parameter matrix, and performs corresponding mirror transformations on the extrinsic rotation relationship and translation horizontal components. When performing proportional scaling enhancement, the training device simultaneously scales the focal length and principal point coordinates. When performing boundary padding enhancement, the training device corrects the principal point coordinates based on the padding offset. When performing cropping enhancement, the training device corrects the principal point coordinates based on the cropping start point and updates the effective image region mask. Only when the image enhancement and the true calibration parameter labels maintain geometric consistency can the trained network learn the true correspondence between pixel space changes and calibration parameter changes.

[0132] Step S203: Generate the standard input tensor for the training phase: The training device performs the same resolution scaling, channel mean alignment, and field-of-view location encoding as in Example 1 on the enhanced historical calibration image to generate the standard input tensor for the training phase. The standard input tensor for the training phase includes an image intensity channel, a normalized horizontal coordinate channel, a normalized vertical coordinate channel, and a radial distance channel. The training device also generates a central field-of-view mask, a transitional field-of-view mask, and an edge field-of-view mask.

[0133] Step S204: Perform the first forward inference and obtain candidate calibration parameters. The training device inputs the standard input tensor from the training phase into the multi-scale feature extraction network to obtain feature tensors at multiple scales. The radial attention fusion module generates scale-related attention weights based on region pooling and global pooling, and fuses them to obtain cross-scale geometric feature vectors. The candidate parameter prediction module outputs candidate intrinsic parameters, candidate rotation parameters, candidate translation parameters, and candidate distortion parameters based on the cross-scale geometric feature vectors.

[0134] Step S205: Generate the projection residual distribution during the training phase and perform secondary forward inference. The training device performs differentiable projection on the 3D spatial reference points corresponding to the training samples based on the candidate calibration parameters to obtain candidate pixel coordinates. The training device compares the candidate pixel coordinates with the real pixel coordinates to generate the projection residual distribution during the training phase. The projection residual distribution during the training phase is input into the feedback weight generation network to generate the feedback weights for the training phase residuals. The multi-scale feature tensor is weighted by the feedback weights for the training phase residuals and then input into the secondary parameter prediction function module to obtain the set of prediction calibration parameters.

[0135] Step S206: Calculate the parameter regression loss. The training device compares the predicted calibration parameter set with the true calibration parameter labels and calculates the parameter regression loss. The parameter regression loss includes intrinsic parameter regression loss, rotation regression loss, translation regression loss, and distortion regression loss. Due to the different dimensions of various parameters, the training device performs scale normalization on different loss terms. Focal length and principal point parameters can be normalized according to image size; translation parameters can be normalized according to calibration space size; rotation parameters can be normalized according to rotation angle or rotation matrix difference; distortion parameters can be normalized according to the pixel offset output by the distortion model.

[0136] Step S207: Calculate the projection consistency loss. The training device projects a 3D spatial reference point based on the predicted calibration parameter set to obtain predicted pixel coordinates. The training device also projects the same 3D spatial reference point based on the true calibration parameter labels to obtain true projected coordinates. The training device calculates the distance between the predicted pixel coordinates and the true projected coordinates, forming the projection consistency loss. The projection consistency loss measures not only whether the values ​​of individual parameters are close, but also whether the combination of parameters is consistent in the imaging result.

[0137] Projection consistency loss can be calculated separately for each field of view region. The projection consistency loss in the central field of view region is used to constrain the principal point and attitude; the projection consistency loss in the peripheral field of view regions is used to constrain distortion and focal length; and the projection consistency loss in the transitional field of view regions is used to constrain overall continuity. The training device can adjust the loss weights for different regions based on the distortion intensity of the samples.

[0138] Step S208: Calculate the distortion monotonic penalty loss. The training device sets multiple radius sampling positions within the effective imaging radius of the normalized image plane and calculates the corresponding distorted radius based on the predicted distortion parameters. If the distorted radii of adjacent radius sampling positions do not satisfy an increasing relationship, the training device accumulates a radius folding penalty. The distortion monotonic penalty loss is used to guide the network to avoid outputting distortion parameters that would cause edge ray folding.

[0139] Step S209: Calculate the positive depth penalty loss. The training device transforms the 3D spatial reference points to the camera's local coordinate system based on predicted rotation and translation parameters, and reads the depth components. If the depth component of a valid reference point is not greater than zero, the training device accumulates a positive depth penalty. The positive depth penalty loss is used to prevent the network output from placing visible reference points behind the camera in the extrinsic parameter combination.

[0140] Step S210: Calculate the feedback effectiveness loss. To ensure that the projected residual distribution is not merely an additional input but actually participates in the secondary prediction improvement, the training device calculates the feedback effectiveness loss. The training device compares the projected residuals corresponding to the first candidate parameters with the projected residuals corresponding to the secondary prediction parameters. If the secondary prediction parameters fail to reduce the error in the residual concentration region, the feedback effectiveness loss increases. The feedback effectiveness loss prompts the feedback weight generation network to learn the correspondence between which residuals should be amplified at which scales, regions, and channels.

[0141] Step S211: Fuse the composite loss according to the dynamic weight strategy. In the early stages of training, the candidate parameter prediction module has not yet formed a stable geometric output. If the weights of the projection consistency loss, distortion monotonic penalty loss, and feedback effectiveness loss are increased too early, the network may have difficulty converging. Therefore, the training device increases the weight of the parameter regression loss in the initial iteration cycle, allowing the network to learn the basic parameter range first. As the projection residual of the validation image decreases, the training device gradually increases the weights of the projection consistency loss, distortion monotonic penalty loss, depth positive penalty loss, and feedback effectiveness loss.

[0142] The dynamic weights related to the training rounds can be increased using a piecewise linear growth method:

[0143]

[0144] in, For the first Class of loss terms in the first Dynamic weights for each training round, dimensionless; For loss item type index, , This represents the total number of loss items. For training rounds, dimensionless; The training epoch where the dynamic weights begin to increase is dimensionless. The training epochs at which the dynamic weights reach their upper limit are dimensionless. For the first The lower bound of the weights of the loss term, dimensionless; For the first The upper limit of the weight of the loss term, dimensionless; This indicates that the training round is in the initial weight preservation phase; This indicates that the training rounds are in the stage of linear weight growth; This indicates that the training rounds are in the phase where the weight cap is maintained.

[0145] The composite loss is formed by a weighted sum of the parametric regression loss, projection consistency loss, distortion monotonicity penalty loss, depth positive value penalty loss, and feedback effectiveness loss. The parametric regression loss maintains a high weight in the early stages of training, while the projection consistency loss, distortion monotonicity penalty loss, depth positive value penalty loss, and feedback effectiveness loss are gradually increased according to the aforementioned dynamic weights. This dynamic weighting strategy first establishes the range of usable parameters during training, and then strengthens projection geometric consistency and physical feasibility.

[0146] Once each loss term enters a stable descent phase, the training device maintains the weight allocation and performs fine-tuning training. Fine-tuning training emphasizes samples from edge view regions, samples with strong distortion, compressed noise samples, and low-light samples. After training, the device solidifies the parameters of the multi-scale feature extraction network, attention mapping network, candidate parameter prediction module, feedback weight generation network, and secondary parameter prediction module, generating a deployable model file.

[0147] Step S212 verifies the physical feasibility of the model output; the training device uses independent test images to verify the model. Verification metrics include intrinsic parameter error, extrinsic parameter error, distortion parameter error, overall reprojection error, edge field-of-view reprojection error, distortion radius folding ratio, depth non-positive reference point ratio, and low confidence image rejection rate. Only when the model meets the deployment conditions on independent test images is the model file sent to the network camera calibration system.

[0148] In this specific implementation, the simulation verification of this embodiment uses 12,000 training images for training, 2,400 verification images for monitoring the training process, and 1,800 independent test images for evaluation. The independent test images include 600 normal field-of-view images, 700 wide-angle field-of-view images, and 500 fisheye approximate field-of-view images. The image resolution is uniformly scaled to 640 x 360 pixels. Each independent test image contains 48 to 80 target pixel coordinates. During the testing process, brightness perturbation, video compression perturbation, and local occlusion perturbation are added. The brightness perturbation ranges from 60% to 140% of the original brightness, the compression quality factor ranges from 25 to 90, and the local occlusion area accounts for no more than 12% of the effective image area.

[0149] Comparative Example 2 uses a single regression scheme with a standard deep residual network, while Comparative Example 3 uses a scheme with projection consistency loss during the training phase but no residual feedback during the inference phase.

[0150] The average reprojection error of the scheme in this embodiment is 0.82 pixels on all 1800 independent test images, compared to 1.46 pixels in Comparative Example 2 and 1.18 pixels in Comparative Example 3.

[0151] The average reprojection error in the edge field of view of this embodiment is 1.09 pixels, compared to 2.37 pixels in Comparative Example 2 and 1.76 pixels in Comparative Example 3. The average rotation angle error of this embodiment is 0.18 degrees, compared to 0.34 degrees in Comparative Example 2 and 0.27 degrees in Comparative Example 3. The relative focal length error of this embodiment is 0.72%, compared to 1.61% in Comparative Example 2 and 1.23% in Comparative Example 3.

[0152] In 500 fisheye approximate field-of-view test images, the scheme in this embodiment did not produce any distortion radius folded samples, while Comparative Example 2 produced 23 distortion radius folded samples, accounting for 4.60% of the 500 fisheye approximate field-of-view test images; Comparative Example 3 produced 9 distortion radius folded samples, accounting for 1.80% of the 500 fisheye approximate field-of-view test images. In the detection of positive depth values ​​at effective reference points, the scheme in this embodiment did not produce any non-positive depth outputs, while Comparative Example 2 produced 11 non-positive depth output samples, and Comparative Example 3 produced 4 non-positive depth output samples.

[0153] In the low-light and compression perturbation subsets, the average reprojection error of the scheme in this embodiment is 1.04 pixels, compared to 1.92 pixels in Comparative Example 2 and 1.51 pixels in Comparative Example 3. In the low-light and compression perturbation subsets, the scheme in this embodiment outputs 122 low-confidence state markers, accounting for 6.78% of the 1800 independent test images; of these, 111 were manually verified to be due to reference point occlusion or target pixel coordinate mismatch, accounting for 90.98% of the low-confidence state marker samples. Comparative Example 2 directly outputs calibration parameters without rejection. Comparative Example 3 outputs 56 low-confidence state markers, accounting for 3.11% of the 1800 independent test images, but some edge-occluded samples were still incorrectly accepted.

[0154] The above data shows that the projection residual distribution feedback enables the secondary prediction to focus on correcting the error concentration area, and the physical feasible region constraint reduces the output of non-physical parameters. This embodiment maintains consistency between the training and inference processes through synchronous geometric enhancement, projection consistency loss, distortion monotonicity penalty loss, depth positive value penalty loss, and feedback effectiveness loss. The residual feedback relationship learned during the training phase can be directly used in the inference phase of Embodiment 1.

[0155] Example 3: This example illustrates the deployment method and anomaly handling mechanism of a network camera field-of-view calibration system based on geometric residual feedback in an edge computing device, as detailed below:

[0156] The network camera field-of-view calibration system based on geometric residual feedback includes a preprocessing module, a multi-scale feature extraction module, a radial attention fusion module, a candidate parameter prediction module, a projection residual feedback module, a quadratic parameter prediction module, and a physical constraint output module.

[0157] The preprocessing module is responsible for video frame reading, image decoding, image quality filtering, resolution scaling, channel mean alignment, field-of-view location encoding, and region mask generation. The preprocessing module outputs a standard input tensor, field-of-view location encoding, region mask, and input metadata. The input metadata includes image size, effective image area, luminance statistics, sharpness statistics, and estimated bitstream compression quality.

[0158] The multi-scale feature extraction module is responsible for executing the multi-scale feature extraction network, extracting feature tensors of multiple scales from the standard input tensor. The multi-scale feature extraction network can use residual convolutional networks, lightweight residual networks, depthwise separable convolutional residual networks, or visual feature networks with residual connections. Regardless of the specific network structure used, the multi-scale feature extraction network needs to output feature tensors of multiple scales while maintaining the correspondence between the field of view position encodings during the downsampling process.

[0159] The radial attention fusion module is responsible for extracting radial partition descriptors for the central, transitional, and edge field-of-view regions based on the region mask, and generating scale-related attention weights. The radial attention fusion module outputs cross-scale geometric feature vectors. It has a feedback data connection with the projection residual feedback module, used to receive residual feedback weights during the secondary prediction stage.

[0160] The candidate parameter prediction module is responsible for outputting candidate intrinsic parameters, candidate rotation parameters, candidate translation parameters, and candidate distortion parameters. This module does not directly generate the final field-of-view calibration parameters. The data output by the candidate parameter prediction module must be fed into the projection residual feedback module.

[0161] The projection residual feedback module is responsible for performing differentiable projection, residual distribution generation, and feedback weight calculation. This module receives a preset spatial reference point and the target pixel coordinates. If there is no manual calibration board on site, the preset spatial reference point can be derived from ground parking space boundaries, road marking endpoints, fixed wall markings, warehouse shelf corners, or on-site structural points determined by measuring equipment. If the network camera is used for recalibration or online verification, the target pixel coordinates can be generated by combining multi-frame structural tracking results and historical reliable calibration parameters, and consistency verification is required using local matching results in the current image.

[0162] The quadratic parameter prediction module is responsible for predicting the calibration parameters to be constrained again based on the feature tensor weighted by residual feedback. This module can output the complete parameters or parameter corrections relative to the candidate calibration parameters. If the parameter correction method is used, the module adds the corrections to the candidate calibration parameters to form the calibration parameters to be constrained. This approach allows the initial prediction result to be used as a benchmark, and the residual feedback is primarily used to correct local biases.

[0163] The physical constraint output module is responsible for executing intrinsic parameter structural constraints, rotational orthogonal constraints, distortion monotonic constraints, depth positive value constraints, confidence gating, and result output. This module also records the constraint correction magnitude. If the correction magnitude for a constraint exceeds the allowable range, it indicates a significant deviation between the network output and the physical model; in this case, the physical constraint output module can improve the low confidence level.

[0164] Regarding exception handling, the system is configured with the following process.

[0165] In the input image anomaly handling process, if the preprocessing module detects that the image is too dark, overexposed, severely blurred, the effective field of view is blocked, or the reference point is not visible, the system will not perform parameter prediction and will directly output the status flag of the reacquisition and calibration image data stream.

[0166] In the projection residual anomaly handling process, if the projection residual generated by the first batch of candidate parameters is abnormally large at a few reference points, the system checks whether the corresponding reference points are occluded or mislabeled. For reference points determined to be abnormal, the projection residual feedback function module reduces the contribution of the corresponding reference point in the feedback weight. If the proportion of abnormal reference points exceeds the allowable proportion, the system outputs a low-confidence status flag.

[0167] In the edge residual centralized processing flow, if the projected residuals are concentrated in the edge field of view region, the system increases the feature weight of the edge field of view region and increases the participation of the distortion prediction branch in the secondary prediction. If the edge residuals still exceed the allowable conditions after the secondary prediction, the system triggers the distortion monotonic constraint and low confidence gating.

[0168] In the depth anomaly handling process, if the depth components of multiple preset spatial reference points in the camera's local coordinate system are not greater than zero, the system first corrects the translation parameters based on the minimum depth offset. If the intrinsic parameters, distortion, and projection residuals are still inconsistent after correction, the system refuses to output the final calibration parameters.

[0169] In the model version exception handling process, if the standard verification tensor fails the output range check after the edge device loads the model file, the system continues to run the old version of the model and records the model loading exception status. This handling can be used for network camera arrays that operate online for extended periods.

[0170] In some preferred embodiments, the field of view ray field is calibrated. The distortion prediction branch does not directly output radial and tangential distortion parameters, but instead outputs the unit field of view ray directions corresponding to multiple radial sampling positions. The physical constraint output module establishes a lookup table of pixels to spatial rays based on the unit field of view ray directions, or fits the unit field of view ray directions to distortion parameters. The projection residual feedback module still generates a projection residual distribution based on the candidate ray field projection error and feeds back a weighted multi-scale feature tensor. This alternative implementation can be used for fisheye lenses or non-standard lenses where the distortion model is difficult to describe with low-order parameters.

[0171] In some preferred embodiments, locally differentiable geometric iteration is employed. After predicting candidate parameters, the system does not immediately proceed to quadratic parameter prediction, but instead performs a finite number of rounds of differentiable geometric optimization near the candidate parameters. Each round of optimization is based on the current parameter projection space reference point to obtain the current projection residual distribution; this distribution is used to update both the parameters and the residual feedback weights. After completing the finite number of rounds of optimization, the system outputs the parameters to be constrained and then enters the physical feasible region constraint. This alternative implementation is suitable for edge servers or central servers with high computing power.

[0172] In the edge deployment verification of this embodiment, the system is deployed on an edge computing box containing an eight-core CPU and a single neural processing unit. The input video stream resolution is 1920 x 1080 pixels, which is scaled down to 640 x 360 pixels during the preprocessing stage. When calibrating a single network camera, the average preprocessing time is 4.6ms, the average time for multi-scale feature extraction and candidate parameter prediction is 18.3ms, the average time for projection residual feedback and secondary prediction is 9.7ms, the average time for physical constraint output is 1.8ms, and the average total time for single-frame calibration is 34.4ms. When using a common single-regression network, the average time for single-frame calibration is 23.6ms. This embodiment reduces the average reprojection error in the edge field of view from 2.37 pixels to 1.09 pixels by increasing the inference time by 10.8ms, making it suitable for calibration tasks where geometric accuracy requirements are higher than single-frame real-time performance.

[0173] In the multi-camera sequential calibration verification, the system continuously processed images from 16 network cameras. Three calibration images meeting the resolution requirements were selected from each network camera. After processing 48 calibration images, the system output four low-confidence status indicators. Manual verification showed that all four low-confidence status indicators corresponded to reference point occlusion or incorrect target pixel coordinate binding. The final field-of-view calibration parameters of the remaining 44 calibration images were fed into the downstream ground landing point calculation module. The average lateral error of the ground projection within the 2-25 meter distance range was 4.8 cm, compared to 8.6 cm in the comparative example three without projection residual feedback. The above edge deployment verification demonstrates that projection residual feedback and physical constraint output not only improve calibration parameter values ​​but also reduce the number of erroneous parameters entering the downstream business process.

[0174] This invention can be deployed on different hardware platforms. For the built-in processor of the network camera, the preprocessing and physical constraint output modules can be executed by the central processing unit, while the multi-scale feature extraction, radial attention fusion, candidate parameter prediction, and secondary parameter prediction modules can be executed by the neural processing unit. For the edge computing box, the video streams from multiple network cameras can sequentially enter a shared model inference queue, and the system maintains independent preset spatial reference points, target pixel coordinates, and historically reliable calibration parameters for each camera. For the video management server, the system can batch process the calibration image data streams from multiple cameras and distribute the final field-of-view calibration parameters to the corresponding cameras or service platforms.

[0175] This invention can also be implemented using a computer-readable storage medium. The computer-readable storage medium stores a computer program. When executed by a processor, the computer program completes the preprocessing, feature extraction, radial attention fusion, candidate parameter prediction, projection residual feedback, secondary prediction, and physical constraint output processes. The computer-readable storage medium can be a solid-state drive, flash memory chip, embedded memory, server disk, or other non-transient storage medium.

[0176] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0177] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for calibrating the field of view of a network camera using a deep residual network, characterized in that, include: The calibration image data stream output by the network camera is obtained, and the target pixel coordinates corresponding to the preset spatial reference point are obtained. The calibration image data stream is subjected to resolution scaling, channel mean alignment and field of view position encoding to generate a standard input tensor containing image intensity information and pixel position constraint information. The standard input tensor is input into a multi-scale feature extraction network with residual connections to extract feature tensors of multiple scales. Scale-related attention weights are generated based on the central region descriptors, edge region descriptors, and global descriptors of each scale feature tensor, and then fused to obtain a cross-scale geometric feature vector. The cross-scale geometric feature vector is input into the parameter prediction network to obtain candidate intrinsic parameters, candidate rotation parameters, candidate translation parameters, and candidate distortion parameters. Based on the candidate intrinsic parameters, candidate rotation parameters, candidate translation parameters, and candidate distortion parameters, a differentiable projection is performed on the preset spatial reference point to generate a projection residual distribution. The feature tensors at the multiple scales are weighted by residual feedback according to the projection residual distribution, and the calibration parameters to be constrained are predicted again based on the feature tensors after residual feedback weighting. The intrinsic parameter structure constraint, rotation orthogonality constraint, distortion monotonicity constraint and depth positive value constraint are applied to the calibration parameters to be constrained to generate the final field of view calibration parameters of the network camera. The generation of the scale-related attention weights includes: dividing the feature tensors of each scale into a central field of view, a transitional field of view, and an edge field of view based on the pixel position constraint information; performing region pooling on the central field of view, the transitional field of view, and the edge field of view respectively to generate a radial partition descriptor; and inputting the radial partition descriptor and the global pooling descriptor into a parameter-shared attention mapping network to output the scale-related attention weights. The residual feedback weighting of the feature tensors at multiple scales includes: resampling the projected residual distribution to the feature plane corresponding to each scale feature tensor according to the downsampling ratio of each scale feature tensor relative to the calibration image data stream; calculating the center residual intensity, transition residual intensity, edge residual intensity, and scale residual intensity in each feature plane; inputting the center residual intensity, the transition residual intensity, the edge residual intensity, and the scale residual intensity into the feedback weight generation network to obtain the residual feedback weights; and using the residual feedback weights to jointly weight the feature tensors at multiple scales in terms of region dimension and channel dimension.

2. The method for calibrating the field of view of a network camera using a deep residual network according to claim 1, characterized in that, The field-of-view position encoding includes: generating a normalized horizontal coordinate channel, a normalized vertical coordinate channel, and a radial distance channel for each pixel position in the calibrated image data stream; and concatenating the normalized horizontal coordinate channel, the normalized vertical coordinate channel, the radial distance channel, and the image intensity channel after aligning the channel mean to obtain the standard input tensor.

3. The method for calibrating the field of view of a network camera using a deep residual network according to claim 1, characterized in that, The multi-scale feature extraction network includes multiple residual convolution stages arranged in series. Each residual convolution stage includes a main path convolutional layer and a residual connection path. The main path convolutional layer extracts spatial features from the input feature tensor. The residual connection path passes the input feature tensor or the input feature tensor after dimension matching to the output of the corresponding residual convolutional stage and adds it element-wise with the feature tensor output by the main path convolutional layer to form a feature tensor of the corresponding scale.

4. The method for calibrating the field of view of a network camera using a deep residual network according to claim 1, characterized in that, Obtaining the candidate intrinsic parameters, candidate rotation parameters, candidate translation parameters, and candidate distortion parameters includes: inputting the cross-scale geometric feature vector into a shared hidden layer; outputting candidate focal length parameters, candidate principal point parameters, and candidate tilt parameters via an intrinsic parameter prediction branch; and filling the candidate focal length parameters, candidate principal point parameters, and candidate tilt parameters into preset matrix positions to construct a candidate intrinsic parameter matrix; outputting candidate rotation vectors and candidate translation vectors via an extrinsic parameter prediction branch; and outputting candidate radial distortion parameters and candidate tangential distortion parameters via a distortion prediction branch.

5. The method for calibrating the field of view of a network camera using a deep residual network according to claim 1, characterized in that, Generating the projection residual distribution includes: transforming the preset spatial reference point to the camera's local coordinate system based on the candidate rotation parameters and the candidate translation parameters; performing normalized imaging processing on the transformed spatial reference point; performing distortion mapping on the normalized imaging processing result based on the candidate distortion parameters; converting the distortion mapping result into candidate pixel coordinates based on the candidate intrinsic parameters; and mapping the coordinate difference between the candidate pixel coordinates and the target pixel coordinates to the corresponding field of view region to obtain the projection residual distribution.

6. The method for calibrating the field of view of a network camera using a deep residual network according to claim 1, characterized in that, The intrinsic parameter structure constraint includes: assembling an intrinsic parameter matrix to be constrained based on the focal length parameter, principal point parameter, and tilt parameter to be constrained in the calibration parameters to be constrained; covering the elements below the main diagonal of the intrinsic parameter matrix to be constrained with zero values; correcting the focal length element to positive values; correcting the scale element to non-zero values; and restricting the principal point parameter to be within the effective pixel range according to the input image size.

7. The method for calibrating the field of view of a network camera using a deep residual network according to claim 1, characterized in that, The rotational orthogonal constraint includes: converting the rotation vector in the calibration parameters to be constrained into an initial rotation matrix, performing singular value decomposition on the initial rotation matrix, constructing a direction correction matrix based on the determinant sign of the product of the left and right singular matrices obtained from the decomposition, and reconstructing the rotation matrix belonging to a special orthogonal group using the left singular matrix, the direction correction matrix, and the right singular matrix.

8. A network camera field-of-view calibration system for deep residual networks, used to execute the network camera field-of-view calibration method for deep residual networks as described in any one of claims 1-7, characterized in that, include: The preprocessing module is used to acquire the calibration image data stream output by the network camera, and to acquire the target pixel coordinates corresponding to the preset spatial reference point. The calibration image data stream is then subjected to resolution scaling, channel mean alignment, and field of view position encoding to generate a standard input tensor containing image intensity information and pixel position constraint information. The multi-scale feature extraction module is used to input the standard input tensor into a multi-scale feature extraction network with residual connections to extract feature tensors at multiple scales. The radial attention fusion module is used to generate scale-related attention weights based on the central region descriptor, edge region descriptor and global descriptor of the feature tensor at each scale, and fuse them to obtain a cross-scale geometric feature vector; The candidate parameter prediction module is used to input the cross-scale geometric feature vector into the parameter prediction network to obtain candidate intrinsic parameters, candidate rotation parameters, candidate translation parameters, and candidate distortion parameters. The projection residual feedback function module is used to perform differentiable projection on the preset spatial reference point based on the candidate intrinsic parameters, the candidate rotation parameters, the candidate translation parameters and the candidate distortion parameters, generate a projection residual distribution, and perform residual feedback weighting on the feature tensors of the multiple scales according to the projection residual distribution; The quadratic parameter prediction module is used to predict the calibration parameters to be constrained again based on the feature tensor after residual feedback weighting. The physical constraint output function module is used to perform intrinsic parameter structural constraints, rotational orthogonal constraints, distortion monotonic constraints, and depth positive constraints on the calibration parameters to be constrained, so as to generate the final field-of-view calibration parameters of the network camera.

Images