An AI-based intelligent optimization shooting adjustment method and a shooting robot

Abstract

The application discloses an AI intelligent optimization shooting adjustment method and a shooting robot, and relates to the technical field of intelligent shooting. In the automatic following shooting process of the high-speed moving shooting subject, a frame rate variable video segment is acquired; the first frame shooting picture and the last frame shooting picture of the frame rate variable video segment are extracted, and the first frame relative position data and the last frame relative position data of the shooting subject are respectively extracted from the first frame shooting picture and the last frame shooting picture; an AI model is used to generate a fitting motion route; a plurality of video adjustment segments are divided in the frame rate variable video segment, and the AI model is used to generate a plurality of corresponding sub fitting routes of the video adjustment segments; each sub fitting route and the fitting motion route are integrated and analyzed, and one or more than one compensation frame picture is inserted into each video adjustment segment. Through the generation of the motion trajectory fitting route and the dynamic frame interpolation compensation of the sub fitting routes of the plurality of video adjustment segments, the video fluency and the picture quality in the high-speed motion scene are improved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of intelligent shooting, and in particular to an AI intelligent optimization shooting adjustment method and a shooting robot. BACKGROUND

[0002] Existing sports shooting videos, especially those related to the shooting of sports ball games, require shooting equipment to have high-precision and high-stability shooting effects. Since the shooting process requires the shooting to capture the players and the ball, the camera equipment needs to move with the players or the ball, which leads to frame loss during shooting, affecting the shooting effect. In particular, in fast-moving scenes, traditional shooting equipment is difficult to capture the complete trajectory of the moving subject in real time, which easily causes problems such as frame freezing and blurring. In addition, the existing technology has a single processing method for frame rate changes, and cannot intelligently compensate according to the actual changes of the moving subject, resulting in uneven video quality. At the same time, the existing shooting robot lacks real-time detection and adaptive adjustment capabilities for frame rate changes, and is difficult to meet the actual needs of professional high-speed motion high-quality real-time shooting. SUMMARY

[0003] The application aims to overcome the shortcomings of the prior art, and provides an AI intelligent optimization shooting adjustment method and a shooting robot. The method includes the steps of: real-time detection of frame rate changes in a shooting process of a high-speed moving subject, generation of a motion trajectory fitting route based on AI model, dynamic frame interpolation compensation based on the sub-fitting routes of the video adjustment segments, effective solution of the problems of frame freezing and incomplete trajectory capture in high-speed motion scenes, and improvement of the video smoothness and picture quality in high-speed motion scenes.

[0004] The application provides an AI intelligent optimization shooting adjustment method, which includes the following steps:

[0005] a. During the automatic following shooting process of a high-speed moving subject, real-time detection of frame rate changes in a shooting picture, and acquisition of a frame rate change video segment;

[0006] b. Extraction of a first frame shooting picture and a last frame shooting picture of the frame rate change video segment, and extraction of first frame relative position data and last frame relative position data of the shooting subject in the first frame shooting picture and the last frame shooting picture, respectively;

[0007] c. Generation of a fitting motion route based on AI model and the first frame relative position data and the last frame relative position data;

[0008] d. Division of the frame rate change video segment into several video adjustment segments, and generation of sub-fitting routes corresponding to the several video adjustment segments based on the AI model;

[0009] e、integrate each segment of the sub-fitting route and the fitting motion route for analysis, and insert one or more compensation frame pictures in each video adjustment segment.

[0010] Further, the step a comprises:

[0011] The detection sliding window is set to detect the real-time shooting pictures in the shooting process in a sliding window manner to obtain detection data.

[0012] The frame rate of the detection data is calculated to obtain a variation index value.

[0013] When the variation index value is greater than a preset detection threshold, the detection data is marked as a frame rate variation video segment.

[0014] Further, the step b comprises:

[0015] b1, selecting a panoramic background picture according to the shooting scene involved in the frame rate variation video segment;

[0016] b2, extracting a shooting subject from the first frame shooting picture and obtaining first frame relative position data of the shooting subject in the panoramic background picture;

[0017] b3, extracting a shooting subject from the last frame shooting picture and obtaining last frame relative position data of the shooting subject in the panoramic background picture.

[0018] Further, the step c comprises:

[0019] c1, analyzing the motion trend of the shooting subject according to the first frame relative position data and the last frame relative data through an AI model;

[0020] c2, inserting a plurality of blank frame pictures between the first frame shooting picture and the last frame shooting picture according to a preset shooting frame rate, and combining the motion trend of the shooting subject to insert a predicted position point of the shooting subject in each blank frame picture;

[0021] c3, constructing a fitting motion route according to the predicted position points of the plurality of blank frame pictures.

[0022] Further, the step c2 comprises:

[0023] Taking the panoramic background picture as the image content of the blank frame, the AI model combines the motion trend of the shooting subject and the relative position data of the shooting subject in the current frame picture to predict the position of the shooting subject in the next frame picture, and obtain a predicted position point of the shooting subject in the next frame picture.

[0024] Further, the step d comprises:

[0025] d1, extract a plurality of shooting references of the frame rate varying video segment, and extract a plurality of process shooting pictures according to the distribution of the shooting references in the frame rate varying video segment;

[0026] d2, divide the frame rate varying video segment into a plurality of video adjustment segments according to the shooting time sequence of the plurality of process shooting pictures;

[0027] d3, obtain the relative position change data of the shooting reference and the shooting subject of each video adjustment segment, and generate a sub-fitting route according to the relative position change data through an AI model.

[0028] Further, the step e includes:

[0029] e1, divide the fitting motion route into a plurality of fitting route segments, and the plurality of fitting route segments correspond to the plurality of fitting sub-routes one by one;

[0030] e2, determine the interpolation compensation position point according to the comparison and analysis of the fitting route segment and the corresponding fitting sub-route;

[0031] e3, generate a motion estimation value according to the fitting route segment, generate a motion sub-data according to the fitting sub-route, and output a compensation frame picture through an AI model according to the motion estimation value and the motion sub-data.

[0032] Further, the step e3 includes:

[0033] According to the shooting content of each fitting route segment, analyze the motion parameters of the shooting subject and the shooting reference;

[0034] Compare the motion parameters with a preset threshold value, and divide the frame interpolation motion level;

[0035] According to the frame interpolation motion level, select a corresponding frame interpolation mode, and generate a compensation frame picture.

[0036] Further, the division operation of the frame interpolation motion level is:

[0037] If , and , the frame interpolation motion level is divided into a high difficulty frame interpolation;

[0038] If , and , the frame interpolation motion level is divided into a low difficulty frame interpolation;

[0039] Wherein, , is a proportionality coefficient, is the motion speed of the shooting subject, is the motion speed of the shooting reference, is the motion direction angle of the shooting subject, To shoot the direction angle of the motion of the reference object.

[0040] The application further provides a shooting robot for AI intelligent optimization of shooting adjustment, characterized in that the shooting robot is used to execute the AI intelligent optimization of shooting adjustment method, and the shooting robot comprises:

[0041] A shooting assembly is used to shoot in real time and transmit the shooting picture to the frame rate processing assembly in real time.

[0042] The frame rate processing assembly is used to detect the frame rate change of the shooting picture and acquire the frame rate change video segment.

[0043] The frame rate compensation assembly is used to analyze the frame picture of the frame rate change video segment and generate the compensation frame picture.

[0044] The application provides an AI intelligent optimization of shooting adjustment method and a shooting robot, which can output high-quality motion shooting video by capturing the state of a shooting subject and a shooting reference object, automatically following the shooting process of the high-speed motion shooting subject, and performing AI intelligent frame compensation operation on the frame drop video segment in combination with the shooting reference object, and can realize efficient video shooting optimization adjustment effect. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a flow chart of the AI intelligent optimization of shooting adjustment method in the embodiment of the application;

[0046] Figure 2 is a method flow chart of step b in the embodiment of the application;

[0047] Figure 3 is a method flow chart of step c in the embodiment of the application;

[0048] Figure 4 is a method flow chart of step d in the embodiment of the application;

[0049] Figure 5 is a method flow chart of step e in the embodiment of the application;

[0050] Figure 6 is a working system schematic diagram of the shooting robot for AI intelligent optimization of shooting adjustment in the embodiment of the application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0052] Embodiment one:

[0053] Figure 1 The flow chart of the AI intelligent optimization shooting adjustment method in the embodiment of the application is shown, which includes steps a to e.

[0054] In the process of automatically following the high-speed moving shooting subject, the frame rate change of the shooting picture is detected in real time, and a frame rate change video segment is obtained.

[0055] A detection sliding window is set to detect the real-time shooting picture in the shooting process, and detection data is obtained.

[0056] The frame rate of the detection data is calculated by standard deviation to obtain a variation index value.

[0057] When the variation index value is greater than a preset detection threshold, the detection data is marked as a frame rate change video segment.

[0058] The window length of the detection sliding window can be 5 to 15 frames, and the window sliding step can be 1 to 3 frames. The window coverage is dynamically adjusted to adapt to the detection needs under different motion speeds. In the standard deviation calculation process, the dispersion degree of the frame rate data in the window is converted into a continuous numerical index by a mathematical formula, for example, the variation index value is calculated by using the sample standard deviation formula. The detection threshold can be dynamically adjusted according to the historical frame rate data, for example, 1.5 to 2.5 times the average value of the historical frame rate standard deviation is taken as the threshold reference. The combination of window sliding and standard deviation calculation effectively captures the spatiotemporal continuity of frame rate fluctuation, avoiding the detection blind area caused by fixed interval sampling.

[0059] Specifically, in the real-time shooting process, the detection sliding window slides along the time axis at a preset step, and the frame rate data of all frames in the window is extracted after each sliding. By calculating the standard deviation of the frame rate in the window, the discrete frame rate fluctuation is converted into a quantitative index, for example, when the window contains 10 frames of data, the standard deviation calculation result can reflect the frame rate stability in the 10 frames. When the variation index value exceeds the dynamically set detection threshold, it is determined that the video segment corresponding to the current window has abnormal fluctuation, for example, in the shooting of a basketball game, when the player suddenly accelerates and the frame rate drops from 60fps to 45fps, the standard deviation in the window significantly increases and triggers the marking. Through the combination of continuous sliding window detection and statistical quantitative analysis, the risk of missing transient fluctuations by single sampling is avoided, and the quantitative index filters the regular fluctuation noise, ensuring the accurate positioning of the abnormal segment.

[0060] Step b, extracting the first frame shooting picture and the last frame shooting picture of the frame rate variation video segment, and extracting the first frame relative position data and the last frame relative position data of the shooting subject in the first frame shooting picture and the last frame shooting picture respectively. Step b is specifically implemented through steps b1-b3.

[0061] b1. Selecting a panoramic background picture according to the shooting scene involved in the frame rate variation video segment;

[0062] When the shooting subject is shot in motion, the panoramic shooting of the motion scene where the shooting subject is located is performed based on the fixed position of the camera unit, so as to obtain the motion shooting scene of the shooting subject, and the corresponding panoramic background picture of the shooting subject in the high-speed motion process can be extracted from the picture data of the panoramic shooting.

[0063] b2. Extracting the shooting subject in the first frame shooting picture and obtaining the first frame relative position data of the shooting subject in the panoramic background picture;

[0064] Specifically, Figure 2 A method flowchart of step b in the embodiment of the application is shown. The shooting subject recognition of the first frame image and the last frame image can be realized through an image recognition algorithm, such as a Faster R-CNN model (Fast Region-based Convolutional Neural Networks), an SSD model (Single Shot MultiBox Detector), a YOLO model (You Only Look Once), and the like, which is not specifically limited here.

[0065] b3. Extracting the shooting subject in the last frame shooting picture and obtaining the last frame relative position data of the shooting subject in the panoramic background picture.

[0066] Further, the shooting subject is also extracted in the last frame shooting picture, and the last frame relative position data of the shooting subject in the panoramic background picture is obtained. In this way, the position data of the first frame and the last frame are both based on the same panoramic background coordinate system, ensuring spatial consistency, and a unified panoramic background reference system is established based on the panoramic background picture, effectively solving the problem of inconsistent coordinate reference in a dynamic scene. By mapping the position of the shooting subject to the panoramic background coordinate system, the reference drift caused by lens movement or changes in the angle of view is eliminated. This double-frame coordinate mapping method based on the panoramic background isolates local picture interference and provides an accurate space-time reference for subsequent motion route fitting, improving the accuracy of motion route fitting.

[0067] Specifically, during the dynamic shooting process, when a frame rate variation video segment is detected, first, a pre-stored panoramic background picture is called as a spatial reference based on the video content features. The first frame picture is aligned with the panoramic background through image registration technology, and the target detection model is used to locate the shooting subject bounding box, calculate the coordinate value of the center point in the panoramic coordinate system, and store it as the starting position. When processing the tail frame, the same panoramic background is reused, and the optical flow method is used to track the subject displacement trajectory, and finally the termination position data based on the unified coordinate system is output. This process eliminates the local picture perspective difference, so that the error of the start and end position data is controlled within 0.5 pixels, providing an accurate space-time reference for subsequent motion route fitting. For example, in football game shooting, even if the camera moves with the player and causes the background to shift, the player's displacement data in the standard field coordinate system can still be accurately obtained through the panoramic stadium map.

[0068] Step c, generating a fitted motion route based on the AI model combined with the first frame relative position data and the tail frame relative position data. Specifically, Figure 3 A method flowchart of step c in the embodiment of the application is shown, including steps c1-c3.

[0069] c1, analyzing the motion trend of the shooting subject according to the first frame relative position data and the tail frame relative position data through the AI model;

[0070] Further, the AI model can be trained based on a neural network model, and the AI model is trained through different sample data of various shooting subjects, so that the AI model can generate the motion route of the shooting subject according to the first frame shooting picture and the tail frame shooting picture.

[0071] c2, inserting a plurality of blank frame pictures between the first frame shooting picture and the tail frame shooting picture according to a preset shooting frame rate, and inserting a predicted position point of the shooting subject in each blank frame picture combined with the motion trend of the shooting subject;

[0072] c3, constructing a fitted motion route according to the predicted position points of the plurality of blank frame pictures.

[0073] Specifically, the step c2 includes: taking a panoramic background picture as the image content of the blank frame, and using the AI model to combine the motion trend of the shooting subject and the relative position data of the shooting subject in the current frame picture to predict the position of the shooting subject in the next frame picture, to obtain the predicted position point of the shooting subject in the next frame picture.

[0074] Specifically, the AI model is configured to analyze the non-linear motion characteristics in the first and last frame position data, such as acceleration or direction mutation, and extract the potential patterns of the motion trajectory through deep learning algorithms. The number of inserted blank frame pictures is set by the actual frame rate of the original video segment, for example, when the target frame rate is 60fps, 58 blank frames need to be inserted between the first and last frame pictures, so that the AI model can compensate for the insertion of frames according to the actual required shooting frame rate, so that the AI model can simulate the motion route of the shooting subject under the preset shooting frame rate.

[0075] Further, the generation of predicted position points uses a time series prediction model, taking the first and last frame positions as input and outputting the position coordinates at intermediate time points. The loss function used in model training can include velocity continuity constraints and acceleration smoothness constraints to generate the motion route of the shooting subject within the continuous frame pictures.

[0076] Specifically, when inserting blank frames between the first and last frames, the timestamps of each blank frame are uniformly distributed according to the preset frame rate, for example, 60 equally spaced time points are inserted within 1 second. The AI model analyzes the position difference between the first and last frames, combines kinematic equations or neural network predictions to predict the position coordinates at intermediate time points, for example, using an LSTM network to capture time-dependent features. During the generation of predicted position points, an error correction mechanism can be introduced, for example, when the velocity change between adjacent predicted points exceeds a threshold, the generation parameters of subsequent predicted points are automatically adjusted. The predicted position points in the blank frame pictures are converted to specific pixel positions in the image through coordinate mapping algorithms, for example, projecting three-dimensional space coordinates to a two-dimensional image plane.

[0077] The construction of the fitted motion route uses a spline interpolation algorithm to connect the discrete predicted position points into a continuous trajectory, where the parameters of the interpolation algorithm are dynamically adjusted according to the motion trend, for example, using a Bezier curve to fit in high-speed rotation scenarios. Thus, by processing the first and last frame data in stages, dynamically generating intermediate predicted points, and optimizing the trajectory fitting algorithm, the accuracy and real-time performance of motion compensation are effectively improved.

[0078] Further, the panoramic background picture is generated by 360-degree scene sampling in the pre-shooting stage, and its resolution is consistent with that of the real-time shooting picture. The AI model uses a convolutional neural network architecture, and the input end receives data packets containing motion trend vectors and current frame relative position coordinates, where the motion trend vector is composed of a velocity component and a direction angle, and the current frame relative position coordinates are based on the panoramic background picture coordinate system.

[0079] During the prediction process, the AI model updates the position reference data in real time at an operation frequency of 30 frames per second, and the fusion weights of the motion trend vector and the position coordinates are dynamically adjusted according to the scene complexity.

[0080] Step d, dividing a plurality of video adjustment segments in the frame rate varying video segment, and generating a plurality of sub-fitting routes corresponding to the plurality of video adjustment segments based on an AI model. Specifically, Figure 4 A method flowchart of step d in the embodiment of the application is shown, which specifically includes steps d1-d3.

[0081] d1, extracting a plurality of shooting reference objects in the frame rate varying video segment, and extracting a plurality of process shooting pictures in the frame rate varying video segment according to the distribution of the shooting reference objects; the extraction of the shooting reference objects can be achieved by using an algorithm based on edge detection or feature point matching, for example, a static object in a scene is identified as a reference object by using a SIFT algorithm.

[0082] d2, dividing the frame rate varying video segment into a plurality of video adjustment segments according to the shooting time sequence of the plurality of process shooting pictures; the extraction of the process shooting pictures can be based on the sampling interval set according to the reference object distribution density, for example, when the reference object distribution density is more than 5 feature points per square pixel, the process shooting pictures are extracted at a rate of 10 frames per second. The division of the video adjustment segments needs to be combined with the time sequence and the reference object distribution characteristics, for example, the continuous segments with a time interval between adjacent process shooting pictures less than 0.5 seconds are combined into one video adjustment segment.

[0083] d3, obtaining the relative position change data of the shooting reference objects and the shooting subject in each video adjustment segment, and generating a sub-fitting route based on the AI model according to the relative position change data.

[0084] The relative position change data can be calculated by using an optical flow method to calculate the displacement vector of the shooting subject and the reference object, and the AI model can use an LSTM network to analyze the time sequence change rule of the displacement vector to generate a sub-fitting route matched with the fitting motion route.

[0085] Specifically, in the frame rate varying video segment, first, the computer vision algorithm is used to identify the object with stable spatial position in the scene as the shooting reference, such as the advertising board or fixed structure in the audience stand in the sports stadium. Based on the spatial distribution density of the reference, the process shooting picture containing the complete distribution characteristics of the reference is extracted at a preset sampling frequency, such as one frame picture is taken every 0.3 seconds. Then, the continuous process shooting pictures are divided into video adjustment segments in time sequence, and the time length of each video adjustment segment can be controlled between 0.5 seconds and 2 seconds. For each video adjustment segment, the relative displacement between the shooting subject and the reference is tracked to generate a dynamic data set containing three-dimensional coordinate changes. The AI model analyzes the direction, speed and acceleration parameters of the displacement vector in the data set to predict the motion trajectory of the shooting subject in the sub-time period, and generates a sub-fitting route that is spatially continuous with the overall fitting motion route. Thus, the division of the video adjustment segment is extended from the pure time dimension to the spatial feature dimension, so that the sub-fitting route can dynamically reflect the cooperative motion law of the shooting subject and the scene reference, and provide accurate spatial position reference for compensating frame insertion.

[0086] Step e, integrating each sub-fitting route and fitting motion route for analysis, and inserting one or more compensation frame pictures in each video adjustment segment. Specifically, Figure 5 A method flowchart of step e in the embodiment of the application is shown, including steps e1-e3.

[0087] e1, dividing the fitting motion route into several fitting route segments, and the several fitting route segments correspond to the several fitting sub-routes one by one;

[0088] e2, determining the frame insertion compensation position point according to the comparison and analysis of the fitting route segment and the corresponding fitting sub-route;

[0089] e3, generating motion estimation value according to the fitting route segment, generating motion sub-data according to the fitting sub-route, and outputting the compensation frame picture through the AI model according to the motion estimation value and the motion sub-data.

[0090] The step e3 specifically includes: analyzing the motion parameters of the shooting subject and the shooting reference according to the shooting content of each fitting route segment. Specifically, the position changes of the shooting subject and the reference in the continuous frames can be tracked by using the computer vision algorithm, so as to obtain the parameters such as motion speed, acceleration and direction. For example, the optical flow method or target tracking algorithm can be used to realize this process.

[0091] The motion parameters are compared with the preset threshold value to divide the frame insertion motion level. A plurality of threshold values can be set to define different motion levels. For example, the motion speed less than 5 pixels / frame can be defined as low-speed motion, 5-15 pixels / frame as medium-speed motion, and greater than 15 pixels / frame as high-speed motion.

[0092] According to the frame filling motion level, a corresponding frame filling mode is selected, and a compensation frame picture is generated.

[0093] The motion parameters can include at least one of a speed and a direction angle, and the preset threshold value can be dynamically adjusted based on different scenes, for example, the speed threshold value is set to 5 m / s in a sports event, and the direction angle difference threshold value is set to 30 degrees. The division of the frame filling motion level can be realized by linear weighting, for example, the weighted sum of the speed of the shooting subject and the shooting reference is compared with the preset threshold value, and when the weighted sum exceeds the threshold value, it is divided into a high difficulty frame filling level. The selection of the frame filling mode can include the switching of the interpolation algorithm type or the generation logic, for example, the optical flow method combined with bidirectional motion estimation is used to generate a compensation frame under a high difficulty level, and a linear interpolation algorithm is used under a low difficulty level.

[0094] Specifically, after integrating the main route and the sub-route, the motion parameters of the shooting subject and the reference are extracted by analyzing the shooting content in each route, for example, the displacement of the shooting subject between consecutive frames is calculated by feature point matching, and the motion trajectory of the reference is obtained by background optical flow analysis. When the motion parameters are compared with the preset threshold value, a multi-dimensional evaluation model is used, for example, different weight coefficients are assigned to the speed and the direction angle respectively, and then a comprehensive score is obtained, and the score result is mapped to three frame filling levels of high, medium and low. According to the level, the frame filling mode is selected, a pre-trained high-precision motion prediction model is called to generate a compensation frame under a high difficulty scene, and the number of compensation frames can be dynamically increased to 1.5 times of the original frame number; a lightweight algorithm based on position interpolation is used to generate a compensation frame under a low difficulty scene. By dynamically matching the frame filling strategy and the motion complexity, the matching error between the compensation frame and the actual motion trajectory can be reduced to within 3 pixels, thereby ensuring the continuity and stability of the video picture in a high-speed motion scene.

[0095] The division operation of the frame filling motion level is:

[0096] If , and , the frame filling motion level is divided into a high difficulty frame filling.

[0097] If , and , the frame filling motion level is divided into a low difficulty frame filling.

[0098] Wherein, , is a proportionality coefficient, is the motion speed of the shooting subject, is the motion speed of the shooting reference, is the motion direction angle of the shooting subject, is the motion direction angle of the shooting reference.

[0099] Further, in the motion change data generation stage, the subject motion trajectory extracted from the first frame and the last frame is quantized as S1 and θ1 parameters, and the reference object motion trajectory extracted from the process frame is quantized as S2 and θ2 parameters. When dividing the motion compensation video segment, the α and β coefficients corresponding to each video segment are dynamically configured according to the scene type.

[0100] Specifically, in the motion compensation video segment processing process, first, the coordinate change sequence of the shooting subject in the continuous frame is extracted through the feature matching algorithm, and the displacement amount of the fixed reference object in the background is detected.

[0101] The shooting subject rate is calculated by dividing the displacement difference between adjacent frames by the time interval, for example, when a football player is detected to move 15 pixels in 0.1 seconds, the motion rate is 150 pixels / second. The direction angle is calculated by the vector angle formed by the coordinate points of the three continuous frames. If the angle change caused by the sudden stop of the basketball player is more than 45 degrees, it is determined that the direction has suddenly changed.

[0102] In the frame interpolation level division stage, the motion parameters of the shooting subject and the reference object are input into the weighted calculation formula, for example, when the motion rate weight α of the player is set to 0.7 and the reference object rate weight β is set to 0.3, the comprehensive motion index exceeds the preset threshold value 8.5, which triggers the high-difficulty frame interpolation mode.

[0103] Further, the frame interpolation mode selection module calls the corresponding algorithm library according to the level identifier, the high-difficulty mode enables the frame interpolation model based on PWC-Net optical flow estimation, and the low-difficulty mode adopts the phase correlation method for motion compensation. By dynamically adjusting the frame interpolation strategy, the video processing system can select the frame interpolation mode flexibly while ensuring the operation efficiency, thereby improving the efficiency of the overall system operation.

[0104] Specifically, the shooting adjustment method further comprises:

[0105] Before the shooting activity, the shooting environment is shot to obtain environment image data;

[0106] The background of each frame of the shooting picture is optimized based on the environment image data.

[0107] The shooting can include selecting multiple sampling points in the shooting scene for multi-angle shooting, and the sampling point spacing can be set to 0.5-1.2 meters, and the shooting number can be 3-5 times. The environment image data can include light intensity, color temperature distribution, and background texture feature parameters.

[0108] Further, the background optimization process can include segmenting the background area in the real-time picture into 8x8 pixel blocks, performing feature comparison with the pre-stored environmental data, triggering noise reduction processing when the matching degree is lower than 85%, and performing optimization processing through brightness compensation, contrast correction, noise reduction intensity adjustment, and the like, so that the shooting video picture can be clear and smooth.

[0109] Specifically, before the shooting activity, the shooting environment can be shot to obtain a panoramic background picture, and each frame of image of the shooting picture is optimized based on the panoramic background picture, so that the overall quality of the shooting video is improved.

[0110] Among them, the shooting can adopt a surround multi-angle shooting or a fixed-point scanning shooting mode, for example, a multi-lens array is mounted on a drone to fly around the shooting scene at 360 degrees, and image data covering the entire scene is collected. After the image data is processed by a three-dimensional reconstruction algorithm, a panoramic background picture is generated, and the resolution can be set to 4K level, and the size of a single picture is controlled within 8000x6000 pixels. In the background optimization process, the dynamic shooting subject and the static background are separated by an image segmentation algorithm, for example, a U-Net neural network model is used for semantic segmentation of each frame of image, and the segmented dynamic subject area is fused with the panoramic background at the pixel level. When the fusion is set, the transparency parameter is set to 0.9 to ensure the natural transition of the subject edge.

[0111] Specifically, before the shooting activity, a wide-angle lens is mounted on the shooting device to shoot the target scene from multiple angles, for example, moving shooting along the top track of the audience seat in the stadium to collect panoramic images covering the game field. During the collection process, the exposure time is set to 1 / 1000 seconds to eliminate dynamic blur, and three groups of photos with different exposure parameters are taken in HDR mode. The image stitching algorithm is used to combine multiple groups of photos into a panoramic background picture, which is stored as a PNG format file with a transparency channel. During subsequent video processing, the background difference method is used to extract the motion subject contour for each frame of image, and the subject contour is mapped to the corresponding coordinate position of the panoramic background. The bilinear interpolation algorithm is used to eliminate the stitching gap. When the compensation frame is inserted, the panoramic background is directly called as the bottom picture, and the complete compensation frame image is generated by combining the predicted trajectory of the moving subject, so as to ensure the spatial consistency of the background elements on the time axis. The scheme establishes a unified background reference library, so that video frames at different time points share the same background data, effectively eliminating the background jump phenomenon caused by changes in light or temporary obstructions.

[0112] The core innovation of the present application is to combine dynamic frame rate detection with multi-level motion trajectory analysis, through the dual mechanism of piecewise fitting and global path integration, to realize the adaptive generation of compensation frames in high-speed motion scenes. This scheme breaks through the limitations of traditional frame interpolation technology relying on fixed algorithms, uses AI models to intelligently predict non-linear motion, effectively eliminates picture stuttering while reducing computational load, and is especially suitable for the optimization of complex motion trajectory shooting such as sports events.

[0113] The working process and principle of the present application are as follows: during the automatic follow-up shooting process of the high-speed moving shooting subject, first, the frame rate variation of the shooting picture is detected in real time to obtain the frame rate variation video segment. This step detects the real-time shooting picture by setting a detection sliding window to obtain detection data, and calculates the standard deviation of the frame rate of the detection data to obtain a variation index value. When the variation index value is greater than a preset detection threshold, the detection data is marked as a frame rate variation video segment.

[0114] Next, the first frame and the last frame of the shooting picture are extracted from the frame rate variation video segment, and the first frame and the last frame of the shooting subject are extracted from the two pictures. The specific operation includes selecting a panoramic background picture according to the shooting scene involved in the frame rate variation video segment, extracting the shooting subject in the first frame and the last frame of the shooting picture, and obtaining the relative position data of the shooting subject in the panoramic background picture.

[0115] Then, based on the AI model, the first frame relative position data and the last frame relative position data are combined to generate a fitted motion route. The AI model first analyzes the motion trend of the shooting subject, inserts a number of blank frame pictures between the first frame and the last frame according to the preset shooting frame rate, and inserts a predicted position point of the shooting subject in each blank frame picture. Finally, the predicted position points are used to construct a fitted motion route.

[0116] Further, several video adjustment segments are divided within the frame rate variation video segment, and the AI model is used to generate a sub-fitted route corresponding to each video adjustment segment. This step includes extracting several shooting reference objects from the frame rate variation video segment, extracting several process shooting pictures according to the distribution of the shooting reference objects, and dividing the video adjustment segments according to the shooting time sequence of the pictures. For each video adjustment segment, the relative position change data of the shooting reference object and the shooting subject is obtained, and the AI model is used to generate a sub-fitted route.

[0117] Finally, each sub-fitted route and the fitted motion route are integrated and analyzed, and one or more compensation frame pictures are inserted into each video adjustment segment. The specific operation includes dividing the fitted motion route into several fitted route segments, which correspond to the sub-fitted routes one by one. The interpolation compensation position points are determined by comparative analysis, the motion estimation values are generated according to the fitted route segments, the motion sub-data are generated according to the fitted sub-routes, and finally the AI model outputs the compensation frame pictures.

[0118] Embodiment two:

[0119] Figure 6 A working system schematic diagram of a shooting robot based on AI intelligent optimization shooting adjustment in an embodiment of the application is shown, the shooting robot is used to execute an AI intelligent optimization shooting adjustment method, and the shooting robot comprises:

[0120] A shooting component 10 is used to shoot in real time and transmit a shooting picture to a frame rate processing component in real time.

[0121] The shooting component 10 can realize real-time picture acquisition by using a CMOS sensor or a CCD sensor, and send original picture data to the frame rate processing component at a transmission rate of no less than 30 fps through a MIPI interface.

[0122] A frame rate processing component 20 is used to detect a frame rate change of a shooting picture and acquire a frame rate change video segment.

[0123] The frame rate processing component 20 can integrate an FPGA chip or a special ASIC processor, after receiving picture data, through a built-in frame rate detection algorithm, the picture stream is scanned once per millisecond, when the standard deviation of the interval time between adjacent frames exceeds a preset threshold, a video segment marking mechanism is triggered immediately, and the abnormal video segment is cached to a DDR4 memory module. The frame rate compensation component can be equipped with a GPU acceleration unit.

[0124] A frame rate compensation component 30 is used to analyze a frame picture of the frame rate change video segment and generate a compensation frame picture.

[0125] The marked video segment is analyzed by using an OpenCL framework, the motion trajectory of a subject and a background in a picture is calculated by using an optical flow method, and an intermediate compensation frame is generated.

[0126] Those skilled in the art can understand that all or part of the steps in the above embodiments can be completed by programs instructing related hardware, the programs can be stored in a computer readable storage medium, and the storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0127] In addition, the above detailed description of the embodiments of the present application is provided, and the principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A shooting adjustment method based on AI intelligent optimization, characterized in that: Including steps: a. When the subject is moving at high speed, real-time detection of frame rate changes in the captured image is performed to obtain the frame rate change video segment; b. extracting the first frame and the last frame of the video segment with a variable frame rate, and extracting the first frame relative position data and the last frame relative position data of the subject from the first frame and the last frame respectively; c. Generate a fitted motion route based on the AI ​​model and the relative position data of the first frame and the last frame; d. Divide the frame rate-changing video segment into several video adjustment segments, and generate sub-fitting routes corresponding to the several video adjustment segments based on the AI ​​model; e. Integrate each sub-fitting route and fitting motion route for analysis, and insert one or more compensation frame images into each video adjustment segment; The step e comprises: e1. Divide the fitted motion route into a number of fitted route segments, where the fitted route segments correspond to the number of fitted sub-routes in a one-to-one manner; e2. Determine the interpolation compensation position point based on the comparative analysis of the fitted route segment and the corresponding fitted sub-route; e3. Generate motion estimation values ​​based on the fitted route segments, generate motion sub-data based on the fitted sub-route, and output compensated frame images based on the motion estimation values ​​and motion sub-data through the AI ​​model; The step e3 comprises: Analyze the motion parameters of the shooting subject and the shooting reference object according to the shooting content of each fitting route segment; Compare the motion parameters with the preset threshold value and classify the interpolated frame motion level; The corresponding frame interpolation method is selected according to the interpolation frame motion level, and a compensation frame image is generated.

2. The AI-based intelligent optimization shooting adjustment method according to claim 1, characterized in that: The step a comprises: Set a detection sliding window to perform sliding window detection on the real-time shooting picture during the shooting process to obtain detection data; Calculating the standard deviation of the frame rate of the detection data to obtain a variation index value; When the change index value is greater than a preset detection threshold, the detected data is marked as a frame rate change video segment.

3. The AI-based intelligent optimization shooting adjustment method according to claim 1, characterized in that: The step b comprises: b1. Select a panoramic background image based on the shooting scene involved in the frame rate change video segment; b2. Extracting the subject from the first frame of the image and obtaining the first frame relative position data of the subject in the panoramic background image; b3. Take a picture at the last frame to extract the subject, and obtain the last frame relative position data of the subject in the panoramic background picture.

4. The AI-based intelligent optimization shooting adjustment method according to claim 1, characterized in that: The step c comprises: c1. Use the AI ​​model to analyze the motion trend of the subject based on the relative position data of the first frame and the relative data of the last frame; c2. Insert several blank frames between the first and last frames according to the preset shooting frame rate, and insert the predicted position of the subject in each blank frame based on the motion trend of the subject; c3. Construct a fitted motion route based on the predicted position points of several blank frame images.

5. The AI-based intelligent optimization shooting adjustment method according to claim 4, characterized in that: The step c2 comprises: The panoramic background image is taken as the image content of the blank frame, and the AI ​​model is used to combine the motion trend of the subject and the relative position data of the subject in the current frame to predict the position of the subject in the next frame, and the predicted position point of the subject in the next frame is obtained.

6. The AI-based intelligent optimization shooting adjustment method according to claim 1, characterized in that: The step d comprises: d1. Extracting several shooting reference objects from the frame rate-varying video segment, and extracting several process shooting pictures from the frame rate-varying video segment based on the distribution of the shooting reference objects; d2. Divide the frame rate change video segment into a number of video adjustment segments according to the shooting time sequence of the plurality of process pictures; d3. Obtain the relative position change data between the shooting reference object and the shooting subject in each video adjustment segment, and generate a sub-fitting route based on the relative position change data through the AI ​​model.

7. The AI-based intelligent optimization shooting adjustment method according to claim 1, characterized in that: The division operation of the interpolation frame motion level is as follows: like ,and , the frame interpolation motion level is classified as high-difficulty frame interpolation; like ,and , classifying the interpolation frame motion level as low difficulty interpolation frame; in, 、 is the proportionality coefficient, is the motion rate of the subject, is the motion rate of the reference object. is the moving direction angle of the subject, The moving direction angle of the reference object.

8. A shooting robot based on AI intelligent optimization shooting adjustment, characterized in that: The shooting robot is used to execute the AI ​​intelligent optimization shooting adjustment method according to any one of claims 1 to 7, and the shooting robot includes: The shooting component is used for real-time shooting and transmitting the shooting images to the frame rate processing component in real time; The frame rate processing component is used to detect frame rate changes in the captured image and obtain frame rate change video segments; The frame rate compensation component is used to perform frame analysis on video segments with frame rate changes and generate compensated frame images.

Images