A site alignment method and device, a terminal device and a storage medium

Abstract

The application relates to the technical field of image processing, and discloses a venue alignment method and device, a terminal device and a storage medium, which are used for reducing the calculation cost and improving the stability of long video venue alignment. The method comprises the following steps: obtaining target semantic evidence representing target venue characteristics from each first video image; obtaining camera parameters corresponding to the first video image based on the target semantic evidence; obtaining global static parameters corresponding to a video stream based on frame static parameters corresponding to multiple first video images; obtaining frame dynamic parameters corresponding to a second video image based on frame dynamic parameters corresponding to a previous video image of the second video image; and projecting a preset standard three-dimensional venue model to the video image based on the frame dynamic parameters of each video image and the global static parameters, so as to obtain a target video image of venue alignment.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a site alignment method, apparatus, terminal device and storage medium. Background Technology

[0002] In intelligent analysis scenarios similar to sports videos, it is necessary to align a pre-set standard 3D field model with the real field in the video image to obtain the precise position of targets such as players and balls in the field coordinate system, providing a foundation for higher-level applications such as tactical analysis and data statistics.

[0003] Current site alignment methods mainly fall into several categories: First, they rely on manual or automatic detection of explicit features such as site corners and lines, establishing mapping relationships through homography or perspective transformation; second, they utilize deep neural networks to directly regress camera parameters or keypoint coordinates; and third, they match site poses from different perspectives using template libraries. These methods face numerous challenges in practical applications. For example, independently optimizing camera parameters frame-by-frame lacks inter-frame constraints, easily leading to result jitter and cumulative drift; performing complete optimization or template retrieval for each frame of a long video incurs high computational costs, making it difficult to meet real-time processing requirements.

[0004] Therefore, how to stably align the site model with the video footage with low computational cost is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, embodiments of this application provide a site alignment method, apparatus, terminal device, and storage medium, which can reduce computational overhead and improve the stability of long video site alignment.

[0006] In a first aspect, embodiments of this application provide a site alignment method, including: Select multiple frames of the first video image from the video stream to be processed; the first video image is a video image in the video stream whose site visibility meets the preset visibility conditions; Obtain target semantic evidence representing the characteristics of the target site from each frame of the first video image; Based on the target semantic evidence corresponding to the first video image, the camera parameters corresponding to the first video image are obtained; the camera parameters include frame static parameters and frame dynamic parameters. Based on the frame static parameters corresponding to the first video images of multiple frames, the global static parameters corresponding to the video stream are obtained; For each frame of the second video image in the video stream other than the first video image, the frame dynamic parameters corresponding to the second video image are obtained based on the frame dynamic parameters corresponding to the previous frame of the second video image. Based on the frame dynamic parameters of each video image and the global static parameters, a preset standard 3D site model is projected onto the video image to obtain a site-aligned target video image.

[0007] In some embodiments, obtaining camera parameters corresponding to the first video image based on target semantic evidence includes: comparing the target semantic evidence corresponding to the first video image with multiple three-dimensional site templates in a preset template library, determining multiple candidate templates that match the first video image from the preset template library; each three-dimensional site template has corresponding camera parameters; determining the matching score between the target semantic evidence and the candidate templates, selecting the candidate template with the highest matching score as the target template; and obtaining the camera parameters corresponding to the first video image based on the camera parameters corresponding to the target template.

[0008] In some embodiments, the target semantic evidence includes occlusion evidence, which is used to identify occluded areas in a video image; determining the matching score between the target semantic evidence and the candidate template, and selecting the candidate template with the highest matching score as the target template, includes: marking the occluded target points as unknown evidence based on the occlusion evidence; ignoring the unknown evidence, and determining the matching score between the target semantic evidence other than the unknown evidence and the candidate template, and selecting the candidate template with the highest matching score as the target template.

[0009] In some embodiments, a candidate range of camera parameters is determined based on the camera parameters corresponding to the target template; the candidate range of camera parameters includes the camera parameters corresponding to the target template; multiple candidate camera parameters are sampled from the candidate range of camera parameters, and feature matching is performed with the target semantic evidence corresponding to the first video image respectively; the candidate camera parameter with the highest matching degree is used as the initial value for optimization and nonlinear optimization is performed, and the optimized camera parameters are used as the camera parameters corresponding to the first video image.

[0010] In some embodiments, obtaining global static parameters corresponding to the video stream based on frame static parameters corresponding to multiple frames of the first video image includes: statistically fusing the same type of frame static parameters in the camera parameters corresponding to multiple frames of the first video image to obtain global static parameters corresponding to the video stream.

[0011] In some embodiments, obtaining the frame dynamic parameters corresponding to the second video image based on the frame dynamic parameters corresponding to the previous frame of the second video image includes: determining a dynamic parameter range based on the frame dynamic parameters corresponding to the previous frame of the second video image; the dynamic parameter range includes the frame dynamic parameters corresponding to the previous frame of the second video image; and determining a target dynamic parameter that minimizes the projection error from the dynamic parameter range as the frame dynamic parameter corresponding to the second video image.

[0012] In some embodiments, the method further includes: calculating the confidence level of dynamic parameters of the current video image; if the confidence level of dynamic parameters is lower than a preset threshold, then updating the frame dynamic parameters of the current video image under the condition that the global static parameters are fixed.

[0013] In some embodiments, target semantic evidence includes at least one of the following: site line evidence, closed region evidence, and target object evidence; site line evidence includes at least one of the following: site line mask, multi-scale grid density features, regional orientation histogram features, and local principal component analysis orientation features; closed region evidence includes at least one of the following: area ratio, centroid location, and area ranking features of the closed region; target object evidence includes at least one of the following: image location, area, and left and right side attributes of the target object in the site.

[0014] Secondly, embodiments of this application provide a site alignment device, comprising: a processing module, configured to select multiple frames of first video images from a video stream to be processed; the first video images are video images in the video stream whose site visibility meets preset visibility conditions; an acquisition module, configured to acquire target semantic evidence characterizing the features of the target site from each frame of the first video image; a pose estimation module, configured to obtain camera parameters corresponding to the first video image based on the target semantic evidence corresponding to the first video image; the camera parameters include frame static parameters and frame dynamic parameters; a static parameter solidification module, configured to obtain global static parameters corresponding to the video stream based on the frame static parameters corresponding to the multiple frames of the first video images; a dynamic parameter following module, configured to obtain frame dynamic parameters corresponding to the second video image for each frame of the second video image other than the first video image in the video stream, based on the frame dynamic parameters corresponding to the previous frame of the second video image; and a projection output module, configured to project a preset standard three-dimensional site model onto the video image based on the frame dynamic parameters of each frame of the video image and the global static parameters, to obtain a target video image for site alignment.

[0015] Thirdly, embodiments of this application provide a terminal device including a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement any of the site alignment methods in the first aspect.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed on a processor, implements any of the site alignment methods in the first aspect.

[0017] The embodiments of this application have the following beneficial effects: By selecting multiple frames of first video images from the video stream to be processed, estimating camera parameters for each frame, and then statistically fusing the frame static parameters of multiple first video images to obtain global static parameters, the random errors in the multi-frame estimation results can be canceled out at the statistical level, thereby obtaining more accurate and stable static parameter estimates, fundamentally avoiding alignment jitter caused by single-frame estimation deviations.

[0018] Building upon this foundation, for each subsequent second video frame, it is no longer necessary to re-estimate all camera parameters. Instead, the frame dynamic parameters of the current video frame are determined using the frame dynamic parameters of the previous frame as initial values, while projection alignment is completed in conjunction with fixed global static parameters. Since the changes in dynamic parameters between adjacent frames are continuous and gradual, using the result of the previous frame as initial values ​​only requires a local search within a small range to determine the dynamic parameters of the current frame. This eliminates the need for a global search frame by frame or large-scale optimization across multiple frames, significantly reducing the computational load per frame and making efficient processing of long videos possible.

[0019] In summary, this application significantly improves the alignment stability of long videos while effectively controlling the computational overhead of single-frame processing, achieving a balance between stability and efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of a site alignment method provided in an embodiment of this application is shown; Figure 2 This illustration shows a schematic diagram of site line evidence for a site alignment method provided in an embodiment of this application; Figure 3 This illustration shows a schematic diagram of a closed area as evidence of a site alignment method provided in an embodiment of this application; Figure 4 This illustration shows a schematic diagram of the target object evidence in a site alignment method provided by an embodiment of this application; Figure 5 This illustration shows a scenario of a site alignment method provided in an embodiment of this application. Figure 1 ; Figure 6 This illustration shows a scenario of a site alignment method provided in an embodiment of this application. Figure 2 ; Figure 7 A schematic diagram of the structure of a site alignment device provided in an embodiment of this application is shown. Detailed Implementation

[0022] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0023] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0024] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0025] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0026] As described in the background section, site alignment schemes suffer from high computational overhead and unstable alignment of long videos.

[0027] Based on this, this application provides a site alignment method. Before initiating the site alignment process, a set of stable global static parameters is obtained using semantic evidence extracted from multiple frames of the first video image. These global static parameters remain unchanged throughout the site alignment process. For each frame of the second video image, a lightweight recursive update is performed only based on the dynamic parameters of the previous frame, without repeatedly solving for the static parameters. Finally, for any given video image, the projection alignment of the standard 3D site model to the image plane is completed by combining the global static parameters with the dynamic parameters of that frame.

[0028] This application decouples camera parameters into two categories: global static parameters and temporal dynamic parameters. Different processing paths are used for estimation and updating of these parameters. This reduces the original high-dimensional, coupled frame-by-frame optimization problem into a one-time global static calibration problem and a low-dimensional temporal dynamic search problem. This significantly reduces the cumulative error and computational cost of long video processing while ensuring the stability of site alignment.

[0029] The site alignment method will now be described in detail with reference to the accompanying drawings and some specific embodiments. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0030] Figure 1 A flowchart of a site alignment method according to an embodiment of this application is shown. This site alignment method can be executed by a terminal device, which may specifically be a server, an extreme machine, or a cloud platform.

[0031] The scene alignment method includes the following steps: S101. Select multiple frames of the first video image from the video stream to be processed.

[0032] The first video image is a video image in the video stream whose site visibility meets the preset visibility conditions.

[0033] Among them, the preset visibility conditions are used to indicate that the visibility of the venue is good. For example, the preset visibility conditions may include: the proportion of the field lines in the video image that are not obscured is greater than a preset threshold (such as 70%), the key areas of the venue (such as the restricted area, the three-point line area, etc.) in the video image are fully visible, and the number of people in the video image is less than a preset number (such as 3 people).

[0034] The video stream to be processed can be a match video in the venue. Each frame of the video stream needs to be aligned with the venue, that is, a preset standard three-dimensional venue model is projected onto each frame in the venue.

[0035] The initial video frames constitute the launch window. Generally, before the start of a match, there are fewer people in the venue, the possibility of obstruction is smaller, and the venue visibility is good. Therefore, the launch window can be preferably the initial stage of the video stream to be processed. Alternatively, the launch window can also be preferably a time period with good visibility in other venues.

[0036] The number of frames N contained in the launch window can be determined by the administrator based on video quality and scene complexity. For example, N can be 5-30, with the first video image covering a video duration of 2-10 seconds.

[0037] Optionally, multiple first video frames are evenly distributed within the startup window.

[0038] S102. Obtain target semantic evidence representing the characteristics of the target site from the first video image of each frame.

[0039] In some embodiments, the target semantic evidence includes at least one of the following: 1. Evidence of site lines, including at least one of the following: site line mask, multi-scale grid density features, regional orientation histogram features, and local principal component analysis orientation features.

[0040] The court line mask is a binary mask image generated by image segmentation or edge detection of the first video image. It is used to mark which pixels in the first video image belong to the court lines (such as sidelines, baselines, center lines, three-point lines, free throw circles, etc.), reflecting the spatial distribution of the court lines.

[0041] Multi-scale grid density features divide an image into grids of multiple scales (e.g., 4×4, 8×8, 16×16), count the number or proportion of field line pixels in each grid cell, and form a density distribution vector. This vector is used to characterize the density of field lines in different regions of the image and to distinguish concentrated areas of field lines from different perspectives (e.g., denser lines in the foreground and sparser lines in the background).

[0042] The regional orientation histogram feature is to statistically analyze the gradient direction of field line pixels in different regions of the image and construct an orientation histogram (e.g., to statistically analyze the response intensity in directions such as 0°, 45°, 90°, and 135°). This is used to capture the main direction of the field lines (horizontal, vertical, and diagonal) and to distinguish different line types (e.g., edge lines are mostly vertical and bottom lines are mostly horizontal).

[0043] Local Principal Component Analysis (PCA) directional features involve performing principal component analysis on the coordinates of sampling points within the local neighborhood of the field line, taking the direction of the first principal component as the dominant direction of that local region, and using it to obtain the precise orientation of the field line at each local location. Compared to the global histogram, it has better spatial resolution and is helpful for matching curved or oblique lines (such as triangular arcs).

[0044] For example, evidence of field lines extracted from images of sporting events can be as follows: Figure 2 As shown.

[0045] 2. Evidence of closed regions, including at least one of the following: the area proportion of the closed region, the location of the centroid, and the area ranking characteristics.

[0046] A playing field typically contains several geometrically or visually closed areas, such as the area inside the three-point line and the free-throw circle in a basketball court, and the free-throw arc and center circle in a soccer field. These closed areas are characterized by fixed shapes, constant relative positions, and stable area proportions. Even if the court lines are partially obscured, as long as fragments of the outlines of these closed areas are identified, the approximate orientation and viewpoint of the camera can still be deduced.

[0047] The area ratio can be the ratio of the pixel area occupied by the enclosed region in the video image to the total area of ​​the entire frame (or the effective area of ​​the field in that frame). The area ratio directly reflects the distance of the region from the camera and its pitch angle. For example, if the enclosed region has a large area ratio, it means the camera is very close to the enclosed region or has a large pitch angle (close-up); if the enclosed region has a small area ratio, it means the camera is far from the enclosed region or has a flat viewing angle (distant view). The area ratio can provide strong constraints on scaling or distance parameters in camera settings.

[0048] The centroid position is used to characterize the geometric center coordinates of the closed region on the image plane. It can reflect the offset direction of the closed region in the picture and is directly related to the camera's yaw angle (horizontal pointing) and lateral displacement, providing a position anchor point for translation or pointing in the camera parameters.

[0049] Area ranking features are used to characterize the order of area size among multiple closed regions in the same frame, to assist in the identification and matching of different closed regions, and to provide a basis for false detection verification. For example, the actual areas of different closed regions in a standard field are fixed and known (e.g., the area of ​​the penalty area > the area of ​​the free throw circle). The order of their projected areas onto the image plane usually remains relatively stable. If the identified area ranking is abnormal (e.g., the area of ​​the free throw circle suddenly becomes larger than that of the penalty area), it indicates that there may be false detection or extreme perspective, and this feature can be used for verification and correction.

[0050] For example, evidence of closed regions extracted from basketball game images can be as follows: Figure 3 As shown, the blue area ① represents the restricted area, the orange-yellow area ② represents the free throw circle, and the green area ③ represents the area inside the three-point line.

[0051] 3. Evidence of the target object, including at least one of the following: the image location, area, and left and right side attributes of the target object in the site.

[0052] A target object refers to a unique and prominent landmark in a playing field. Target objects are generally independent and constant in terms of spatial, geometric, and scale recognition dimensions. Examples include backboards, hoops, baskets, and 24-second shot clocks in a basketball court; goals and corner flags in a soccer field; and net posts and service markers in a tennis court.

[0053] Object evidence, utilizing image observation information of landmarks in the field, can provide supplementary constraints for camera parameter estimation that are independent of the field line, especially when the field line is occluded.

[0054] Image position refers to the two-dimensional pixel coordinates of the target object in the video image. It is usually taken as the center point of the target object detection box or a key point on the target object (such as the center of the basketball hoop, the midpoint of the bottom edge of the backboard, the midpoint of the crossbar of the goal, etc.). It can reflect the geometric relationship between the camera's line of sight and the target object, and is used to provide direct position anchoring constraints for the yaw angle (horizontal rotation angle) and pitch angle (vertical rotation angle) in the camera parameters.

[0055] Area can be the pixel area occupied by an object in a video image, such as the area of ​​the object detection box or the number of pixels in the object's outline region. According to the principle of perspective projection, the area of ​​the same object in an image is inversely proportional (approximately) to the square of its distance from the camera. For example, a large object area indicates that the object is close to the camera, suggesting a large equivalent focal length or shallow depth; a small object area indicates that the object is far from the camera, suggesting a small equivalent focal length or great depth. Area can provide strong constraints on scaling or depth in camera parameters, especially when the field lines are severely occluded and distance cannot be inferred from line length, making object area a key basis for distance estimation.

[0056] Left / right attributes identify whether a target object is located on the left or right side of the target area (such as a basketball court). These attributes can be determined by assessing the target object's horizontal position relative to the court area, using evidence from court lines (such as the detected center line or restricted area position) for further analysis, or employing temporal search (the target object's movement path across consecutive frames). Left / right attributes eliminate matching ambiguities, ensuring that detected targets correctly correspond to unique targets in the standard 3D court model.

[0057] For example, target evidence extracted from basketball game images can be as follows: Figure 4 As shown, the image is divided into an 8×8 grid, where the red ① represents the target object—the backboard, the orange ② represents the target object—the blue frame, and the green ③ indicates that the target object is hit in that grid, while the other grids are not hit.

[0058] Preferably, the target semantic evidence may also include occlusion evidence, which is used to identify occluded areas in the video image.

[0059] Occlusion evidence is essentially marking untrusted areas in the field line mask. Field line sampling points or field line pixels that fall within the occlusion area are marked as "unknown" or "invalid". This actively excludes these interfered evidences during subsequent camera parameter estimation, thus avoiding the introduction of erroneous constraints.

[0060] The source of obstruction can be people or objects on the field. For example, players, the ball, referees, coaches, advertising boards, camera shadows, etc.

[0061] Obtaining occlusion evidence can be specifically implemented as follows: detecting occlusions in the first video image and generating an occlusion mask; obtaining the site line sampling points of the first video image; and marking the sampling points within the coverage area of ​​the occlusion mask as unknown. Site line sampling points marked as unknown are not involved in the subsequent camera parameter determination process to eliminate occlusion interference.

[0062] S103. Based on the target semantic evidence corresponding to the first video image, obtain the camera parameters corresponding to the first video image.

[0063] The camera parameters for video images include frame static parameters and frame dynamic parameters. Static parameters are camera parameters that remain unchanged during video acquisition, while dynamic parameters are camera parameters that change during video acquisition.

[0064] Specifically, static parameters may include at least one of the following: camera pitch angle, camera roll angle, camera lateral position, camera distance from the edge line, and camera height. Dynamic parameters may include camera yaw angle and / or camera focal length scaling parameters.

[0065] In some embodiments, step S103 may specifically be implemented as follows: comparing the target semantic evidence corresponding to the first video image with multiple three-dimensional site templates in a preset template library, and determining multiple candidate templates that match the first video image from the preset template library; each three-dimensional site template has corresponding camera parameters; determining the matching score between the target semantic evidence and the candidate templates, and selecting the candidate template with the highest matching score as the target template; and obtaining the camera parameters corresponding to the first video image based on the camera parameters corresponding to the target template.

[0066] The preset template library is generated by sampling within a discretized parameter space based on a 3D model of a standard target site. This parameter space covers camera yaw, zoom, pitch, roll, height, and lateral position. For example, yaw is uniformly sampled at 1-5 degree intervals within the 0-360 degree range; zoom is discretely sampled at preset step sizes within a continuous range covering typical lens focal lengths; pitch is uniformly sampled at preset intervals within the -30 to 30 degree range; camera height is discretely sampled at preset step sizes within the 3-15 meter range; and camera lateral position is discretely sampled at preset step sizes within a preset range along the target site's edge.

[0067] For each parameter combination, the corresponding projection matrix, site line mask, and closed area mask are pre-calculated to form candidate templates. Each candidate template pre-stores the site model projection features and corresponding camera parameters under the corresponding parameter combination.

[0068] Yaw represents the camera's rotation angle around its vertical axis, used to control the camera's horizontal viewing angle. Pitch represents the camera's rotation angle around its horizontal axis, used to control the camera's vertical pitch angle. Roll represents the camera's rotation angle around its optical axis (shooting direction). Zoom represents the change in the camera lens's focal length, affecting the distance and size of objects in the image.

[0069] Optionally, the total storage size of the preset template library is approximately 100MB, with 3800 candidate templates. Each template carries complete camera extrinsic parameters (camera position, attitude angle) and intrinsic parameters (focal length, principal point offset, etc.), and all are line template images, which are efficient in storage and fast in retrieval.

[0070] Each template is pre-calculated and stored during generation, containing its corresponding handcrafted feature vectors (such as the line length and orientation distribution of the template). These handcrafted feature vectors can specifically include image block features (multi-scale grid density), partition histogram features (orientation distribution), region area features (area ratio of closed regions), and vector field signatures for lines and curves.

[0071] Optionally, before template matching, candidate templates in a pre-defined template library can be pre-filtered based on the target object visibility judgment in the target semantic evidence and the initial yaw angle prior, in order to narrow the search scope to a subset of candidates matching the current viewpoint and improve template retrieval efficiency. For example, some irrelevant templates can be pre-filtered based on whether the backboard is visible and the yaw prior.

[0072] This application categorizes the target semantic evidence corresponding to the first video image into three independent evidence families: site line evidence, closed area evidence, and target object evidence. These families are input into an evidence fusion and consistency scoring mechanism, which calculates the matching score between each evidence family and each candidate template. When multiple evidence families consistently support a candidate template, the candidate template's ranking significantly improves; conversely, when multiple evidence families conflict, the candidate template's ranking decreases. Based on this consistency assessment, the fused matching score is output. The candidate templates are then ranked according to their matching scores, and multiple candidate templates are selected as an initial candidate set.

[0073] The matching score can be calculated based on indicators such as grid density similarity and region orientation similarity.

[0074] In some embodiments, the process of determining the matching score between the target semantic evidence and the candidate template is as follows: First, feature similarity calculations are performed on the site line evidence, closed region evidence, and target object evidence extracted from the first video image. Specifically, the site line evidence score is obtained by comparing the similarity of grid density feature vectors; the closed region evidence score is obtained by comparing the intersection and overlap ratio of the detection region and the template region, as well as the distance between their center points; and the target object evidence score is obtained by comparing the relative distance between the target object image position and the template projection position, combined with the left and right side attribute matching. Then, preset weighting weights are assigned to the site line score, closed region score, and target object score, respectively. Each score is multiplied by its corresponding weight and summed to obtain the total matching score of the candidate template. Finally, the candidate templates in the preset template library are traversed, and the candidate template with the highest total matching score is selected as the target template.

[0075] In some embodiments, determining the matching score between the target semantic evidence and the candidate template, and selecting the candidate template with the highest matching score as the target template, can be specifically implemented as follows: based on occlusion evidence, the occluded target point is marked as unknown evidence; the unknown evidence is ignored, and the matching score between the target semantic evidence other than the unknown evidence and the candidate template is determined, and the candidate template with the highest matching score is selected as the target template.

[0076] By ignoring evidence points within the occluded area and only performing matching and scoring based on valid evidence that is not occluded, the robustness of the template matching process in occluded scenarios is further enhanced.

[0077] In some embodiments, obtaining the camera parameters corresponding to the first video image based on the camera parameters corresponding to the target template can be specifically implemented as follows: determining a candidate range of camera parameters based on the camera parameters corresponding to the target template; the candidate range of camera parameters includes the camera parameters corresponding to the target template; sampling multiple candidate camera parameters from the candidate range of camera parameters, and performing feature matching with the target semantic evidence corresponding to the first video image respectively; using the candidate camera parameter with the highest matching degree as the initial value for optimization and performing nonlinear optimization, and using the optimized camera parameters as the camera parameters corresponding to the first video image.

[0078] Centered on the camera parameters corresponding to the target template, new candidates are sampled within the neighborhood of its yaw and zoom levels for more refined feature matching. Nonlinear optimization is performed within a constrained parameter space, fine-tuning all camera degrees of freedom to minimize the matching error between the projected 3D model lines and the image evidence.

[0079] Nonlinear optimization can employ iterative optimization algorithms (such as Levenberg-Marquardt) to fine-tune camera parameters. The optimization objective is to minimize projection errors, including: the distance error between the projected model lines and the field line mask in the target semantic evidence; the intersection-over-union (IoU) error between the projected region and the detected region in the target semantic evidence; and the Euclidean distance between the projected target position and the detected target position in the target semantic evidence. During the optimization process, the variation range of each camera parameter is constrained within a preset physically reasonable range (e.g., within a certain neighborhood of the initial values) to avoid getting trapped in local optima or generating physically unreasonable solutions.

[0080] In some embodiments, the process of nonlinearly optimizing camera parameters includes: constructing a comprehensive optimization objective and using an iterative optimization algorithm to adjust camera parameters, including yaw angle, pitch angle, roll angle, focal length scaling, camera height, and lateral position, to minimize the comprehensive projection error. The comprehensive projection error is composed of multiple accumulated errors, specifically including: the distance deviation error between the 3D field line projected onto the image plane and the actually detected field line mask; the overlap loss between the 3D closed region projection patch and the actual detected region patch; and the pixel distance error between the 3D target object projection position and the actual detected target object image position. During the optimization process, corresponding proportional balance coefficients are introduced for each error term, and penalty constraints are added to address the deviation of camera parameters from the initial template parameters. This ensures that the optimization algorithm can converge stably when fine-tuning camera parameters and ultimately outputs camera parameters that best fit the model projection to the image features.

[0081] In other embodiments, step S103 can be implemented through direct geometric analytical solution. Specifically, geometric feature points can be extracted from the acquired target semantic evidence (such as court line masks, closed area outlines, or target object positions) to identify multiple two-dimensional key points with clear three-dimensional physical space coordinates in the first video image, such as the intersection of the court sidelines and baselines, the corner of the penalty area, or feature points of fixed targets (such as goalposts or backboard reference points). Subsequently, these extracted two-dimensional key points are paired one by one with the corresponding known three-dimensional space key points in the preset standard three-dimensional court model to construct a set of point-to-point mapping relationships from the two-dimensional plane to the three-dimensional space. Then, the perspective projection matrix corresponding to the image is solved directly using the Perspective-n-Point (PnP) algorithm or the Direct Linear Transformation (DLT) algorithm. Based on matrix factorization technology, the camera parameters corresponding to the first video image, which include frame static parameters (such as pitch angle, altitude, etc.) and frame dynamic parameters (such as yaw angle, focal length, etc.), are extracted. This direct geometric solution method eliminates the need to pre-build and store a large 3D site template library, enabling rapid calculation of single-frame camera parameters with extremely low computational power consumption. It provides feasible technical support for site alignment processing in resource-constrained terminal devices or simple scenarios.

[0082] S104. Based on the frame static parameters corresponding to the first video image of multiple frames, obtain the global static parameters corresponding to the video stream.

[0083] In some possible implementations, step 104 can be achieved through simple arithmetic mean fusion or weighted linear fusion based on frame quality factors. Specifically, after extracting multiple first video images from the startup window of the video stream to be processed and calculating the frame static parameters corresponding to each first video image, the arithmetic mean of the frame static parameters of the same type in the multiple first video images is directly calculated, and the arithmetic mean is used as the global static parameter corresponding to the video stream to be processed; or, the frame static parameters corresponding to the multiple first video images are weighted and normalized according to the quality assessment results of each first video image (such as weight coefficients determined based on image sharpness, illumination uniformity, or site feature extraction confidence), and the resulting weighted average is used as the global static parameter.

[0084] Preferably, in some embodiments, step S104 can be specifically implemented as: statistically fusing the same type of frame static parameters in the camera parameters corresponding to multiple frames of the first video image to obtain the global static parameters corresponding to the video stream to be processed.

[0085] Statistical fusion methods can include: calculating the median, calculating the weighted median, truncated mean, robust estimation (RANSAC), etc.

[0086] For example, obtain the set of pitch angles of multiple frames of the first video image within the startup window, perform statistical fusion on the set of pitch angles, and use the statistical fusion result as the pitch angle value in the global static parameters.

[0087] By employing the aforementioned statistical fusion method, random errors in the frame static parameters corresponding to multiple frames of the first video image can be statistically canceled out, resulting in more accurate and stable global static parameter estimates. These global static parameters remain fixed and uniformly applied throughout the subsequent video stream processing. Compared to directly using the estimation result of a single frame or independently estimating all parameters frame by frame, this method fundamentally avoids inter-frame fluctuations in static parameters caused by single-frame estimation deviations, providing a reliable parameter basis for stable alignment of subsequent video frames.

[0088] S105. For each frame of the second video image in the video stream other than the first video image, obtain the frame dynamic parameters corresponding to the second video image based on the frame dynamic parameters corresponding to the previous frame of the second video image.

[0089] Among them, the video image preceding the earliest second video image is the first video image of the last frame in the multi-frame first video image.

[0090] In some embodiments, step S105 may be specifically implemented as follows: determining a dynamic parameter range based on the frame dynamic parameters corresponding to the previous frame of the second video image; the dynamic parameter range includes the frame dynamic parameters corresponding to the previous frame of the second video image; and determining the target dynamic parameter that minimizes the projection error from the dynamic parameter range as the frame dynamic parameter corresponding to the second video image.

[0091] The dynamic parameter range is the preset neighborhood range of the dynamic parameters of the frame corresponding to the previous frame of video image.

[0092] For example, the preset neighborhood range can be within ±5 degrees of the estimated value of the yaw in the previous frame, and within ±10% of the estimated value of the zoom in the previous frame. This can cover the maximum variation of camera dynamic parameters between adjacent frames under normal shooting conditions, while keeping the search space within a sufficiently small range, thus balancing search accuracy and computational efficiency.

[0093] Since the changes in image content between adjacent frames in a video image are continuous and gradual, the camera yaw angle and focal length scaling parameters do not change abruptly between adjacent frames. Therefore, by using the dynamic parameter values ​​estimated from the previous frame as the initial values ​​for the current frame search, and performing a local search within a preset neighborhood of these initial values, the optimal dynamic parameters for the current frame can be determined without having to search again in the global parameter space, thus significantly reducing the computational load per frame.

[0094] In some embodiments, after calculating the dynamic parameters of the current video image, the confidence level of the dynamic parameters of the current video image is also calculated. If the confidence level of the dynamic parameters is lower than a preset threshold, the frame dynamic parameters of the current frame video image are updated under the condition that the global static parameters are fixed.

[0095] The confidence level of dynamic parameters can be determined based on line alignment score (the degree of matching between the projected field lines and the field line mask extracted from the image), visible point ratio (the proportion of unoccluded field model projection points to the total projection points), and motion smoothness prior (the degree of difference between the dynamic parameters of the current frame and the dynamic parameters of the previous frame). The line alignment score, visible point ratio, and motion smoothness prior distribution reflect the reliability of the current dynamic parameter estimation results from three dimensions: projection accuracy, occlusion degree, and motion continuity.

[0096] Specifically, the average pixel deviation between the projected lines of the 3D site model and the actual site lines in the image is calculated; the smaller the deviation, the higher the line alignment. The proportion of model projection points falling within the unobstructed area of ​​the image to the total number of projection points is calculated as the effective point visibility ratio. Simultaneously, the changes in the estimated yaw angle and focal length scaling values ​​of the current frame are compared with those of the previous frame; the smoother the changes, the higher the motion smoothness. The line alignment, effective point visibility ratio, and motion smoothness are then weighted and combined according to a preset ratio to obtain a dynamic parameter confidence value between zero and one.

[0097] If the confidence level of the dynamic parameters is lower than the preset threshold (e.g., lower than 0.6), it indicates that the dynamic parameter estimation result of the current frame may have a large error. In this case, under the condition that the global static parameters are fixed, the dynamic parameters of the current frame video image are re-searched, that is, a semi-global search is performed within the preset search range of Yaw and Zoom to update the frame dynamic parameters of the current frame video image.

[0098] By re-searching and determining dynamic parameters in this way, we can effectively deal with abnormal situations such as rapid camera switching or local search failures, and ensure the accuracy of tracking subsequent frames.

[0099] S106. Based on the frame dynamic parameters of each video image and the global static parameters, the preset standard three-dimensional site model is projected onto the video image to obtain the site-aligned target video image.

[0100] Specifically, based on the frame dynamic parameters and global static parameters corresponding to the current frame video image, the complete camera state of the frame is determined; based on the complete camera state, the 3D projection matrix and ground homography matrix are calculated; the preset standard 3D site model is projected onto the image plane of the current frame video image to generate the site-aligned target video image and alignment confidence score.

[0101] The standard 3D court model is a pre-constructed 3D geometric model of the target court. For example, taking a basketball court as an example, the standard 3D court model can include precise geometric parameters such as: 28m × 15m sidelines, a three-point line arc with a radius of 6.75m, a 5.8m × 4.9m restricted area rectangle, and the basket (1.575m from the baseline, 3.05m high). In this example, when extracting evidence, attention should be paid to distinguishing between the three-point line arc and the center circle (e.g., by using the different positions of their curvature centers), as well as backboard detection (as a target object with strong spatial constraints). Double verification can also be performed in the static parameters based on the geometric relationship between the projected height of the basket and its actual height of 3.05 meters.

[0102] For different venues, such as standardized sports fields like football fields, volleyball courts, and tennis courts (e.g., the penalty area, center line, and corner area of ​​a football field), it is only necessary to replace the corresponding standard three-dimensional venue model and adjust the evidence extraction method to identify the unique lines and regional features of the venue (e.g., the goal area and penalty arc of a football field). Other operation steps do not need to be changed. The venue alignment scheme provided in this application has a wide range of applications.

[0103] In actual projection, the projection results can be adjusted based on bundle adjustment (BA) so that the total error between the projected position of the 3D model and the detected position in the real image is minimized.

[0104] A single template matching operation takes about 1 millisecond, and a complete single-frame processing, including bundle adjustment optimization, takes about 1-2 seconds. Site alignment takes less time.

[0105] Figure 5 and Figure 6 Two scene diagrams of the field alignment process are given, where (a) is the original event image, (b) is the image from which target semantic evidence is extracted for template matching, (c) is the template projection image matched based on (b), and (d) is the target video image of the field alignment obtained by projecting (c) onto (a).

[0106] The court alignment method provided in this application can be applied to automated analysis of sports videos. Taking basketball shooting event analysis as an example, its application chain is as follows: 1. Event Trigger: An intersection between the ball and the backboard is detected, which is determined to be a shooting event; 2. Frame backtracking: Backtrack to the frame of the ball throwing point to obtain the player's position on the court before the shot; 3. Site alignment: The site alignment scheme provided in this application is used to align the standard 3D site model with the image in this frame; 4. Coordinate mapping: Based on the alignment results, calculate the precise position of the shooter in the court coordinate system; 5. Score Determination: Based on the geometric relationship between the shooting position and the three-point line, the score of the shot (2 points or 3 points) is automatically determined.

[0107] No real-time scene rendering is required; only 1-2 key event frames need to be processed in each match video to complete the field coordinate mapping.

[0108] The site alignment solution provided in this application is described below from an overall perspective: 1. Input sports videos; 2. Select to start the window for N frames; 3. Extract multiple types of semantic evidence; 4. Roughly recall a portion of candidate templates from the template library; 5. Based on multiple evidence families, candidate templates are re-ranked for consistency, and the target template with the highest score is selected; 6. Perform local Yaw / Zoom fine-tuning and adjustment of the static parameters of the target template; 7. Determine the static parameters of each frame in the startup window; 8. Determine global static parameters based on the static parameters of multiple frames within the startup window; 9. Using the dynamic parameters of the previous frame as initial values, optimize the dynamic parameters Yaw / Zoom of the current frame; 10. Generate the projection matrix based on the optimized dynamic parameters; 11. Output the 3D alignment result.

[0109] In summary, the site alignment scheme provided in this application achieves the following significant benefits through a two-stage strategy of "start-up window self-calibration + static parameter solidification" and "dynamic parameter tracking," as well as multi-evidence family fusion and occlusion-friendly mechanisms: 1. Alignment stability is fundamentally improved.

[0110] This scheme selects multiple first-frame video images from the video stream to be processed, performs fully free-degree-of-freedom, high-cost camera parameter estimation on each, and then fixes the static parameters in the multi-frame estimation results through cross-frame statistical fusion. Since various interference factors (such as occlusion, lighting changes, etc.) may exist in a single frame image, the static parameters estimated independently in a single frame often contain random errors. If directly used for subsequent frame processing, this would cause the court model to jitter around the actual location. This scheme, through multi-frame statistical fusion, ensures that the estimation results of normal frames are dominant in quantity and effectively suppresses the influence of outliers, thereby obtaining more accurate and stable global static parameters, which remain fixed in the processing of all subsequent frames. Experimental results show that in basketball game videos with tens of thousands of frames, the inter-frame jitter of the court line projection is reduced by more than 70%, fundamentally eliminating the problem of cumulative drift of static parameters in long videos, and achieving an order-of-magnitude improvement in alignment stability.

[0111] 2. Computational efficiency is greatly improved.

[0112] After locking the global static parameters, this solution determines the dynamic parameters of the current frame for each second video image frame only based on the dynamic parameters of the previous frame, without needing to re-estimate all camera parameters frame by frame. Since the changes in camera dynamic parameters between adjacent frames are continuous and gradual, using the result of the previous frame as an initial value, the parameter determination for the current frame can be completed by performing a local search within a preset neighborhood, eliminating the need for a large-scale search in the global parameter space. In practical tests, this solution reduces the computational complexity of each frame from a global search related to the template library size to a constant-level local search, improving overall video processing speed by 5-8 times, reducing computational load by 65%-80%, and decreasing the average CPU / GPU computing resources required per frame by approximately 70%. The lightweight dynamic parameter search enables this solution to complete processing in near real-time (with latency within several frames), making fully automated long video analysis feasible and facilitating deployment on edge devices or large-scale video processing platforms.

[0113] 3. The initial recall accuracy of the template is significantly improved.

[0114] This scheme introduces three independent evidence families—field lines, enclosed areas, and target objects—in the template matching stage. The matching degree between each evidence family and the candidate template is calculated separately. The ranking of a candidate pose is significantly improved only when multiple evidence families consistently support the same candidate pose. This multi-evidence family consistency fusion decision avoids the mismatch problem caused by severe occlusion or similar line shapes (such as the third line arc and the center circle) when relying solely on field lines for matching. Experimental results show that even in the extreme case where occlusion renders more than 50% of the field lines invisible, the initial pose recall accuracy remains above 85%, an improvement of over 30 percentage points compared to single-evidence methods.

[0115] 4. Significantly enhanced resistance to shading.

[0116] This solution extracts occlusion evidence and marks sampling points falling into occlusion areas as unknown evidence for omission during matching scoring. The matching score is calculated only based on visible evidence that is not occluded. By actively marking and ignoring evidence in occlusion areas, the algorithm can still function normally when players cover 50% of the court area and can still output effective alignment results in densely populated areas under the basket, significantly expanding its applicability.

[0117] In summary, the field alignment method provided in this application, while maintaining high alignment accuracy, takes into account both the stability and computational efficiency of long video processing through the comprehensive effect of four mechanisms: static parameter fixing of the startup window, dynamic parameter following, multi-evidence family fusion, and occlusion-friendly processing. This provides an efficient and reliable foundation for the automated analysis of sports videos.

[0118] Based on the description of the above method embodiments, the terminal device for executing the site alignment method provided in this application should include the following functional modules: 1. The video preprocessing and frame sampling module receives the raw video stream, performs decoding and basic color space conversion, is responsible for keyframe extraction or uniform time sampling, and selects the start window frame sequence for self-calibration based on a strategy (such as the video start position). Its input is a video file or video stream, and its output is an RGB image frame sequence and a start window (first video image) identifier.

[0119] 2. A multi-class semantic evidence extraction module is used to extract target semantic evidence from each input frame of the image. Specifically, this module runs multiple lightweight neural networks or traditional vision algorithms in parallel to extract evidence such as site lines, closed regions, target objects, and occlusion. Its input is a single-frame RGB image, and its output is a structured evidence dictionary.

[0120] 3. The 3D template library and management module is used to store and retrieve pre-generated 3D site templates. It uses a spatial index data structure (such as a KD-tree) to organize the parameter space and supports fast approximate nearest neighbor search based on feature vectors to select target templates. Its input is the evidence feature vector, and the output is a list of candidate templates sorted by similarity and their corresponding camera parameters.

[0121] 4. The parameter optimization and estimation module is used to achieve consistency scoring of multiple evidence families and fine-grained matching of local neighborhoods; to perform multi-degree-of-freedom optimization based on minimizing reprojection error; to perform robust statistics on the optimization results of multiple frames within the startup window to estimate and lock global static parameters; to perform lightweight dynamic parameter optimization based on motion priors under the condition that global static parameters are fixed; and to calculate the dynamic parameter confidence of the current frame and trigger a re-search process when the dynamic parameter confidence falls below a preset threshold. Its inputs include evidence, candidate templates, and camera parameters from the previous frame, and its outputs are the optimized camera state, projection matrix, and dynamic parameter confidence.

[0122] 5. The projection and output module is used to calculate the 3D-to-2D projection matrix and ground plane homography matrix based on the final camera parameters. It projects all elements (lines, surfaces, and points) of the standard 3D site model to image coordinates and serializes the alignment results for each frame, including the projected line drawing, region fill map, and metadata file containing all parameters. Its input is the camera parameters, and its output is the projection matrix, the alignment result visualization image, and the data file.

[0123] Figure 7 A schematic diagram of a field alignment device according to an embodiment of this application is shown. Exemplarily, the field alignment device 200 includes a processing module 201, an acquisition module 202, an attitude estimation module 203, a static parameter solidification module 204, a dynamic parameter tracking module 205, and a projection output module 206.

[0124] The system includes the following components: a processing module 201, which selects multiple frames of first video images from the video stream to be processed; the first video images are video images in the video stream whose site visibility meets preset visibility conditions; an acquisition module 202, which acquires target semantic evidence representing the target site features from each frame of the first video image; a pose estimation module 203, which obtains camera parameters corresponding to the first video image based on the target semantic evidence corresponding to the first video image; the camera parameters include frame static parameters and frame dynamic parameters; a static parameter solidification module 204, which obtains global static parameters corresponding to the video stream based on the frame static parameters corresponding to multiple frames of the first video image; a dynamic parameter following module 205, which obtains the frame dynamic parameters corresponding to the second video image for each frame of the second video image other than the first video image in the video stream based on the frame dynamic parameters corresponding to the previous frame of the second video image; and a projection output module 206, which projects a preset standard three-dimensional site model onto the video image based on the frame dynamic parameters of each frame of the video image and the global static parameters to obtain a site-aligned target video image.

[0125] In some embodiments, the pose estimation module 203 is specifically used to: compare the target semantic evidence corresponding to the first video image with multiple three-dimensional site templates in a preset template library, and determine multiple candidate templates that match the first video image from the preset template library; each three-dimensional site template has corresponding camera parameters; determine the matching score between the target semantic evidence and the candidate templates, and select the candidate template with the highest matching score as the target template; and obtain the camera parameters corresponding to the first video image based on the camera parameters corresponding to the target template.

[0126] In some embodiments, the target semantic evidence includes occlusion evidence, which is used to identify occluded areas in a video image; the pose estimation module 203 is specifically used to: mark the occluded target points as unknown evidence based on the occlusion evidence; ignore the unknown evidence, and determine the matching score between the target semantic evidence other than the unknown evidence and the candidate template, and select the candidate template with the highest matching score as the target template.

[0127] In some embodiments, the pose estimation module 203 is specifically used to: determine a candidate range of camera parameters based on the camera parameters corresponding to the target template; the candidate range of camera parameters includes the camera parameters corresponding to the target template; sample multiple candidate camera parameters from the candidate range of camera parameters, and perform feature matching with the target semantic evidence corresponding to the first video image respectively; use the candidate camera parameter with the highest matching degree as the initial value for optimization and perform nonlinear optimization, and use the optimized camera parameters as the camera parameters corresponding to the first video image.

[0128] In some embodiments, the static parameter solidification module 204 is specifically used to: statistically fuse the static parameters of the same type of frames corresponding to multiple frames of the first video image to obtain the global static parameters corresponding to the video stream.

[0129] In some embodiments, the dynamic parameter following module 205 is specifically used to: determine a dynamic parameter range based on the frame dynamic parameters corresponding to the previous frame of the second video image; the dynamic parameter range includes the frame dynamic parameters corresponding to the previous frame of the second video image; and determine the target dynamic parameter that minimizes the projection error from the dynamic parameter range as the frame dynamic parameter corresponding to the second video image.

[0130] In some embodiments, the dynamic parameter following module 205 is further configured to: calculate the confidence level of the dynamic parameters of the current video image; if the confidence level of the dynamic parameters is lower than a preset threshold, then update the frame dynamic parameters of the current video image under the condition that the global static parameters are fixed.

[0131] It is understood that the device in this embodiment corresponds to the site alignment method in the above embodiment, and the options in the above embodiment are also applicable to this embodiment, so they will not be described again here.

[0132] This application also provides a terminal device, exemplary of which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the terminal device to perform the functions of the various modules in the above-described site alignment method or site alignment device.

[0133] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0134] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0135] This application also provides a computer-readable storage medium for storing the computer program used in the aforementioned terminal device. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0137] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0138] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0139] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A site alignment method, characterized in that, include: Select multiple frames of the first video image from the video stream to be processed; The first video image is a video image in the video stream whose site visibility meets a preset visibility condition; Obtain target semantic evidence characterizing the features of the target site from each frame of the first video image; Based on the target semantic evidence corresponding to the first video image, the camera parameters corresponding to the first video image are obtained; the camera parameters include frame static parameters and frame dynamic parameters. Based on the frame static parameters corresponding to the multiple frames of the first video image, the global static parameters corresponding to the video stream are obtained; For each frame of the second video image in the video stream other than the first video image, the frame dynamic parameters corresponding to the second video image are obtained based on the frame dynamic parameters corresponding to the previous frame of the second video image. Based on the frame dynamic parameters of each video image and the global static parameters, a preset standard three-dimensional site model is projected onto the video image to obtain a site-aligned target video image.

2. The method according to claim 1, characterized in that, The step of obtaining camera parameters corresponding to the first video image based on the target semantic evidence corresponding to the first video image includes: The target semantic evidence corresponding to the first video image is compared with multiple three-dimensional site templates in a preset template library, and multiple candidate templates matching the first video image are determined from the preset template library; each three-dimensional site template has corresponding camera parameters; Determine the matching score between the target semantic evidence and the candidate template, and select the candidate template with the highest matching score as the target template; Based on the camera parameters corresponding to the target template, the camera parameters corresponding to the first video image are obtained.

3. The method according to claim 2, characterized in that, The target semantic evidence includes occlusion evidence, which is used to identify occluded areas in a video image; The step of determining the matching score between the target semantic evidence and the candidate template, and selecting the candidate template with the highest matching score as the target template, includes: Based on the occlusion evidence, the occluded target point is marked as unknown evidence; Ignore the unknown evidence, determine the matching score between the target semantic evidence other than the unknown evidence and the candidate template, and select the candidate template with the highest matching score as the target template.

4. The method according to claim 2, characterized in that, The step of obtaining the camera parameters corresponding to the first video image based on the camera parameters corresponding to the target template includes: Based on the camera parameters corresponding to the target template, a candidate range of camera parameters is determined; the candidate range of camera parameters includes the camera parameters corresponding to the target template. Multiple candidate camera parameters are sampled from the candidate range of camera parameters, and feature matching is performed between each candidate parameter and the target semantic evidence corresponding to the first video image. The candidate camera parameters with the highest matching degree are used as the initial values ​​for nonlinear optimization, and the optimized camera parameters are used as the camera parameters corresponding to the first video image.

5. The method according to claim 1, characterized in that, The step of obtaining the global static parameters corresponding to the video stream based on the frame static parameters corresponding to the multiple frames of the first video image includes: Statistical fusion is performed on the static parameters of the same type of frame in the camera parameters corresponding to the multiple frames of the first video image to obtain the global static parameters corresponding to the video stream.

6. The method according to claim 1, characterized in that, The process of obtaining the frame dynamic parameters corresponding to the second video image based on the frame dynamic parameters corresponding to the previous frame of the second video image includes: Based on the frame dynamic parameters corresponding to the previous frame of the second video image, a dynamic parameter range is determined; the dynamic parameter range includes the frame dynamic parameters corresponding to the previous frame of the second video image. The target dynamic parameter that minimizes the projection error is determined from the range of dynamic parameters and used as the frame dynamic parameter corresponding to the second video image.

7. The method according to claim 1, characterized in that, The method further includes: Calculate the confidence score of the dynamic parameters of the current video image; If the confidence level of the dynamic parameter is lower than a preset threshold, then the frame dynamic parameters of the current video image are updated under the condition that the global static parameters are fixed.

8. The method according to claim 1, characterized in that, The target semantic evidence includes at least one of the following: site line evidence, closed area evidence, and target object evidence; The site line evidence includes at least one of the following: site line mask, multi-scale grid density features, regional orientation histogram features, and local principal component analysis orientation features; The evidence of the closed region includes at least one of the following: the area ratio of the closed region, the location of its centroid, and the area ranking characteristics; The evidence of the target object includes at least one of the following: the image location, area, and left and right side attributes of the target object in the site.

9. A site alignment device, characterized in that, include: The processing module is used to select multiple frames of the first video image from the video stream to be processed; The first video image is a video image in the video stream whose site visibility meets a preset visibility condition; The acquisition module is used to acquire target semantic evidence representing the characteristics of the target site from each frame of the first video image; The pose estimation module is used to obtain camera parameters corresponding to the first video image based on the target semantic evidence corresponding to the first video image; the camera parameters include frame static parameters and frame dynamic parameters. The static parameter fixing module is used to obtain the global static parameters corresponding to the video stream based on the frame static parameters corresponding to the multi-frame first video images; The dynamic parameter following module is used to obtain the frame dynamic parameters corresponding to the second video image for each second video image in the video stream other than the first video image, based on the frame dynamic parameters corresponding to the previous video image of the second video image. The projection output module is used to project a preset standard three-dimensional site model onto the video image based on the frame dynamic parameters of each video image and the global static parameters, so as to obtain a site-aligned target video image.

10. A terminal device, characterized in that, The terminal device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the site alignment method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed on a processor, implements the site alignment method according to any one of claims 1-8.

Images