Dual-lens motion camera switching method, device and computer equipment

Through the combination of intelligent collaborative model and micro electronically controlled liquid crystal optical components, the delay and error problems in the switching process of dual-lens motion cameras are solved, real-time and seamless lens switching is achieved, and image quality and user experience are improved.

CN118574002BActive Publication Date: 2024-11-12SHENZHEN ZHENCHENG TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411048167.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-12
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

The existing dual-lens sports cameras have problems such as delay, picture jitter, and discontinuous exposure during the lens switching process, making it difficult to accurately grasp the best switching timing, affecting the user experience and video quality.

Method used

Through the configuration of light transmission paths, scene feature extraction of historical video data and intelligent collaborative model construction, real-time video data is obtained for frame-level importance weight map analysis, combined with dynamic threshold judgment and reinforcement learning, the lens switching strategy optimization is achieved, and through micro-electric The optical path adjustment of the liquid crystal optical element ensures smoothness and image quality.

Benefits of technology

It improves the smoothness and image quality of lens switching, reduces mechanical errors, realizes real-time and seamless lens switching, and improves the intelligence and user experience of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118574002B_ABST
    Figure CN118574002B_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of camera switching, and discloses a switching method, device and computer equipment for a dual-lens motion camera. The method includes: configuring the light transmission path of the dual-lens motion camera, obtaining the target optical path and collecting historical video data; extracting scene features from the historical video data, obtaining scene feature data and building a lens intelligent collaboration model; acquiring real-time video data and performing real-time frame-by-frame analysis of the video sequence to obtain a frame-level importance weight map; performing multi-dimensional feature extraction and time series modeling through the lens intelligent collaboration model to obtain a lens switching probability prediction curve; performing threshold judgment and reinforcement learning analysis to obtain the optimal lens switching control strategy; and performing progressive adjustment of the target optical path through the optimal lens switching control strategy and generating target video data. The present application improves the smoothness and image quality of dual-lens switching.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of camera switching, and in particular to a switching method, device and computer equipment for a dual-lens motion camera. Background Art

[0002] With the rapid development of mobile devices and Internet technology, action cameras have become an important tool for recording outdoor activities and extreme sports. Traditional single-lens action cameras often face problems such as limited viewing angles and fixed focal lengths during shooting, making it difficult to meet users' diverse shooting needs in different scenarios. To solve this problem, dual-lens action cameras have emerged, integrating wide-angle and telephoto lenses to provide more flexible shooting options. However, current dual-lens action cameras still face many challenges during lens switching, such as switching delays, image jitter, and discontinuous exposure, which seriously affect users' shooting experience and video quality.

[0003] In addition, the existing dual-lens switching technology mainly relies on manual operation or simple scene recognition algorithms, which makes it difficult to accurately grasp the best switching time. This not only increases the user's operating burden, but also makes it easy to miss wonderful moments. At the same time, traditional optical systems often use mechanical structures for adjustment during the lens switching process. This method not only has a slow response speed, but also easily introduces mechanical errors and affects image quality. On the other hand, existing image processing algorithms often find it difficult to achieve real-time and smooth switching effects when processing fast-moving scenes, resulting in obvious faults or distortions in the video.

[0004] Therefore, there is an urgent need for a dual-lens motion camera switching method that can intelligently predict the switching timing, achieve high-speed seamless switching, and ensure image quality. Summary of the invention

[0005] The present application provides a switching method, device and computer equipment for a dual-lens motion camera, and the present application improves the smoothness and image quality of dual-lens switching.

[0006] In a first aspect, the present application provides a method for switching a dual-lens motion camera, the method comprising:

[0007] Configure the light transmission path for the dual-lens motion camera, obtain the target optical path and collect historical video data;

[0008] Extracting scene features from the historical video data to obtain scene feature data and constructing a lens intelligent collaboration model;

[0009] Acquire real-time video data and perform real-time frame-by-frame analysis of video sequences to obtain a frame-level importance weight map;

[0010] The intelligent lens collaboration model is used to perform multi-dimensional feature extraction and time series modeling on the frame-level importance weight map to obtain a lens switching probability prediction curve;

[0011] Performing threshold judgment and reinforcement learning analysis on the shot switching probability prediction curve to obtain an optimal shot switching control strategy;

[0012] The target optical path is progressively adjusted through the optimal lens switching control strategy to generate target video data.

[0013] In a second aspect, the present application provides a switching device for a dual-lens motion camera, the switching device for a dual-lens motion camera comprising:

[0014] A configuration module is used to configure the light transmission path of the dual-lens motion camera, obtain the target optical path and collect historical video data;

[0015] An extraction module, used to extract scene features from the historical video data, obtain scene feature data and construct a shot intelligent collaboration model;

[0016] An analysis module is used to obtain real-time video data and perform real-time frame-by-frame analysis of video sequences to obtain a frame-level importance weight map;

[0017] A modeling module, used for performing multi-dimensional feature extraction and time series modeling on the frame-level importance weight map through the lens intelligent collaborative model to obtain a lens switching probability prediction curve;

[0018] A judgment module, used to perform threshold judgment and reinforcement learning analysis on the shot switching probability prediction curve to obtain an optimal shot switching control strategy;

[0019] An adjustment module is used to progressively adjust the target optical path through the optimal lens switching control strategy and generate target video data.

[0020] A third aspect of the present application provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the above-mentioned dual-lens motion camera switching method.

[0021] In the technical solution provided by the present application, the real-time dynamic adjustment of the light transmission path is realized through the precise control of the micro-electrically controlled liquid crystal optical elements, which greatly improves the flexibility and response speed of the optical system and effectively reduces the delay and error caused by the traditional mechanical switching. The constructed lens intelligent collaboration model can analyze and predict complex scenes in real time, accurately grasp the best switching time, and improve the degree of automation and shooting effect. The deep feature extraction and temporal relationship modeling of video data improve the accuracy of switching probability prediction. Combined with dynamic threshold judgment and reinforcement learning analysis, the adaptive optimization of the switching strategy is realized, and the switching behavior can be automatically adjusted according to different scenes and user habits, which improves the intelligence and user experience of the system. Through Bezier curve interpolation and real-time ray tracing technology, the smooth transition of the optical path is realized, and the jitter and distortion of the picture during the switching process are effectively reduced. Using dynamic refractive index distribution calculation and real-time correction matrix, combined with multi-frame super-resolution reconstruction technology, the image quality and detail performance during the switching process are significantly improved. By calculating the exposure compensation coefficient and color space conversion in real time, the exposure continuity and color consistency between different lenses are ensured, and the smoothness and image quality of dual-lens switching are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0023] Figure 1 A schematic diagram of an embodiment of a switching method of a dual-lens motion camera in an embodiment of the present application;

[0024] Figure 2 Schematic diagram of an embodiment of a switching device for a dual-lens motion camera in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application provide a switching method, device and computer equipment for a dual-lens motion camera. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 , an embodiment of a method for switching a dual-lens motion camera in an embodiment of the present application includes:

[0027] Step 101: configure the light transmission path of the dual-lens motion camera to obtain the target optical path and collect historical video data;

[0028] It is understandable that the execution subject of the present application may be a switching device of a dual-lens motion camera, or a terminal or a server, which is not limited here. The present application embodiment is described by taking a server as the execution subject as an example.

[0029] Specifically, the initial voltage is set for the micro-electrically controlled liquid crystal optical element of the dual-lens motion camera to obtain the initial light refractive index distribution, which reflects the propagation characteristics of light in the liquid crystal optical element. The optical path of the incident light is calculated according to the initial light refractive index distribution to obtain the initial light transmission path, and the optical simulation analysis is performed on this path to obtain relevant optical performance indicators. Through optical simulation analysis, the performance of the optical system under the current configuration is determined. According to the optical performance indicators, the voltage distribution of the micro-electrically controlled liquid crystal optical element is iteratively optimized, and the optical performance is improved by continuously adjusting the voltage to obtain the optimal voltage control scheme. Through the voltage control scheme, an optimized target optical path is generated to ensure that the transmission efficiency and image quality of light in the optical element are maximized. After the target optical path is generated, the exposure parameters of the image sensor of the dual-lens motion camera are set to obtain the initial exposure configuration. Based on the initial exposure configuration, the shooting environment is sampled in multiple scenes to obtain a sample video data set. The sample video data set is used for subsequent image quality evaluation, and the image quality score in each scene can be obtained through the evaluation. According to these scores, the initial exposure configuration is adaptively adjusted to obtain the optimized exposure configuration. Based on the optimized exposure configuration and target optical path, video capture of multiple typical scenes is performed. The optimized optical path and exposure configuration can ensure the best video quality in different shooting environments.

[0030] Step 102: extract scene features from historical video data to obtain scene feature data and build a lens intelligent collaboration model;

[0031] Specifically, through spatiotemporal segmentation, historical video data is divided into multiple spatiotemporal blocks, so that feature extraction can be performed at a fine granularity. Multi-scale convolutional neural network processing is used to extract the primary feature map of each spatiotemporal block. In order to improve the expressive power of the feature map, the attention mechanism is applied, and the weighted feature map is obtained through weighted processing. Non-local correlation analysis is performed to capture global context features, so as to understand the relationship between various elements in the scene in a larger range. Multiple spatiotemporal blocks are clustered based on global context features to obtain scene type labels. Through these labels, the temporal relationship of the spatiotemporal blocks is modeled, and a scene transition probability matrix is ​​established. A Markov decision process model is constructed based on the scene type label and the scene transition probability matrix to obtain the initial state transfer function. Through Monte Carlo tree search, an optimized state value function is obtained. Based on this optimized function, a deep reinforcement learning network is constructed to form a shot coordination strategy network. In order to ensure the effectiveness of the strategy network, multiple rounds of strategy iteration and experience playback training are performed on it. This process can continuously optimize the strategy network, making it perform better in practical applications. In this embodiment, an efficient intelligent lens coordination model is finally obtained, which can extract scene features and predict lens switching based on historical video data, thereby achieving more intelligent and automated lens switching.

[0032] Step 103, acquiring real-time video data and performing real-time analysis of the video sequence frame by frame to obtain a frame-level importance weight map;

[0033] Specifically, real-time video data of a dual-lens motion camera is obtained, and the real-time video data is decoded in parallel to obtain an original frame sequence. Through parallel decoding, a large amount of video data can be efficiently processed to ensure the real-time and accuracy of subsequent analysis. Optical flow estimation is performed on the original frame sequence to obtain a motion vector field between adjacent frames. Optical flow estimation can capture motion information between video frames and help understand dynamic changes in the scene. The original frame sequence is filtered in the time domain according to the motion vector field to obtain a denoised frame sequence. Temporal filtering helps to remove random noise between frames, improve the quality of the frame sequence, and ensure the stability and reliability of subsequent processing steps. Multi-scale pyramid decomposition is performed on the denoised frame sequence to obtain multi-level image features, which can reflect the information of the image at different scales. A deep residual network is constructed based on the multi-level image features to generate a frame-level semantic feature map. The deep residual network can capture high-level semantic information in the image through feature extraction and fusion at multiple levels. The self-attention mechanism is calculated on the frame-level semantic feature map to obtain a key area heat map. The self-attention mechanism can identify key areas in the image and enhance the features of these areas through weighted processing to obtain an enhanced feature map. The enhanced feature map is processed by graph convolutional network to obtain the local-global relationship matrix. Graph convolutional network can effectively model the complex relationship between pixels in the image, especially the relationship between local and global. According to the local-global relationship matrix, the importance score of each pixel is calculated to obtain the frame-level importance weight map. The frame-level importance weight map can reflect the importance of each pixel in each frame and provide a basis for subsequent shot switching decisions.

[0034] Step 104: Perform multi-dimensional feature extraction and time series modeling on the frame-level importance weight map through the lens intelligent collaborative model to obtain a lens switching probability prediction curve;

[0035] Specifically, a three-dimensional convolution operation is performed on the frame-level importance weight map to obtain a spatiotemporal feature tensor. The three-dimensional convolution operation can simultaneously capture the features of video data in the spatial and temporal dimensions. The spatiotemporal feature tensor is weighted to obtain weighted spatiotemporal features, highlighting key features so that the model can more accurately identify important spatiotemporal information. After obtaining the weighted spatiotemporal features, the autoencoder is used to compress them to obtain a compressed feature representation. The autoencoder effectively reduces the dimension of the features through the encoding and decoding process while retaining important information. A variational inference network is constructed based on the compressed feature representation to obtain feature distribution parameters. The variational inference network can capture the intrinsic distribution of features by probabilistically modeling the features. The feature distribution parameters are sampled through the lens intelligent collaborative model to obtain multiple groups of feature samples. These feature samples can be used to train a Gaussian process regression model to obtain a probability prediction function. The Gaussian process regression model can effectively predict the changing trend and uncertainty of the features by performing regression analysis on the samples, and provide reliable probability prediction for lens switching. After the probability prediction function is trained, it is subjected to Bayesian optimization to obtain the optimal hyperparameters. Bayesian optimization can find the parameter combination that optimizes model performance by probabilistic modeling and optimization of hyperparameters. Based on the optimal hyperparameters, the probability prediction function is subjected to sequence rolling prediction and curve construction to obtain the shot switching probability prediction curve. Sequence rolling prediction can dynamically adjust the shot switching strategy by gradually predicting future feature changes, making the switching process smoother and more intelligent.

[0036] Step 105: Perform threshold judgment and reinforcement learning analysis on the shot switching probability prediction curve to obtain the optimal shot switching control strategy;

[0037] Specifically, the shot switching probability prediction curve is subjected to wavelet transform to obtain multi-scale frequency features. Wavelet transform can decompose different frequency components of the signal, thereby revealing the potential patterns and features in the switching probability curve. According to the multi-scale frequency features, the dynamic threshold is calculated to identify possible shot switching moments and obtain the initial switching moment set. The initial switching moment set is clustered to identify candidate switching intervals in a practical sense. Through clustering analysis, similar switching moments are clustered together to form a more representative switching interval. According to the candidate switching intervals, a state transfer matrix is ​​constructed. The state transfer matrix describes the transition probability between different states. After constructing the state transfer matrix, the initial policy network is generated and the corresponding reward function is designed. The reward function is used to evaluate the pros and cons of each policy and provide guidance for the reinforcement learning process. By combining the initial policy network with the reward function, a reinforcement learning environment is established. In this environment, the policy network gradually improves its decision-making ability through continuous trial and adjustment. The reinforcement learning environment is subjected to proximal policy optimization to obtain an optimized policy network. Proximal policy optimization is an efficient reinforcement learning method that ensures the stability and continuity of the policy by limiting the amplitude of each update. Based on the optimized policy network, multiple groups of switching policies are generated to form a policy sample set. The policy sample set is cross-validated to evaluate the performance of each policy. Cross-validation ensures the universality and robustness of the policy by testing on different data sets. According to the performance evaluation indicators, the policy sample set is sorted and screened to finally obtain the optimal lens switching control strategy.

[0038] Step 106: progressively adjust the target optical path through the optimal lens switching control strategy and generate target video data.

[0039] Specifically, the adjustment sequence of the target optical path is calculated according to the optimal lens switching control strategy to obtain a segmented adjustment scheme. The segmented adjustment scheme is to gradually adjust the optical path in different time periods to ensure the smoothness and continuity of the switching process. The segmented adjustment scheme is interpolated with Bezier curves to obtain a smooth transition path. Bezier curve interpolation can effectively smooth the transition path so that no abrupt changes will occur during the adjustment of the optical path, thereby ensuring the stability and continuity of the video image. According to the smooth transition path, the micro-electrically controlled liquid crystal optical element is voltage modulated to obtain a dynamic refractive index distribution. Voltage modulation accurately controls the refractive index of the optical element so that it can adapt to different light transmission paths and achieve dynamic adjustment. The dynamic refractive index distribution is simulated by ray tracing to obtain an instantaneous optical characteristic curve. Ray tracing simulation can accurately calculate the propagation path and change law of light in the optical system, thereby providing optical characteristic data. According to the instantaneous optical characteristic curve, the image distortion compensation parameter is calculated to obtain a real-time correction matrix. The image distortion compensation parameter is used to correct the image distortion caused by the change of the optical path to ensure that the geometric shape and proportion of the video image remain correct. Bilinear interpolation is performed on the real-time correction matrix to obtain a pixel-level mapping relationship. Bilinear interpolation is a commonly used image processing technology. By interpolating the correction matrix, the mapping relationship of each pixel can be obtained to ensure the accuracy and delicacy of the image transformation. According to the pixel-level mapping relationship, a geometric transformation is performed to obtain a corrected image sequence. Geometric transformation is to rearrange and adjust the image to meet the expected geometric shape and proportion. Multi-frame super-resolution reconstruction is performed on the corrected image sequence to obtain a high-resolution image sequence. Multi-frame super-resolution reconstruction can significantly improve the resolution and detail performance of the image by combining the information in multiple frames of images. After obtaining the high-resolution image sequence, the exposure compensation coefficient and color space conversion parameters are calculated to ensure the brightness and color consistency of the image. The exposure compensation coefficient is used to adjust the brightness of the image to keep it consistent under different lighting conditions; the color space conversion parameters are used to adjust the color of the image to meet the expected color performance. Finally, after exposure compensation and color space conversion, the target video data is obtained.

[0040] In the embodiment of the present application, the real-time dynamic adjustment of the light transmission path is realized through the precise control of the micro-electrically controlled liquid crystal optical element, which greatly improves the flexibility and response speed of the optical system and effectively reduces the delay and error caused by the traditional mechanical switching. The constructed lens intelligent collaboration model can analyze and predict complex scenes in real time, accurately grasp the best switching time, and improve the degree of automation and shooting effect. The deep feature extraction and temporal relationship modeling of video data improve the accuracy of switching probability prediction. Combined with dynamic threshold judgment and reinforcement learning analysis, the adaptive optimization of the switching strategy is realized, and the switching behavior can be automatically adjusted according to different scenes and user habits, which improves the intelligence and user experience of the system. Through Bezier curve interpolation and real-time ray tracing technology, the smooth transition of the optical path is realized, and the picture jitter and distortion during the switching process are effectively reduced. Using dynamic refractive index distribution calculation and real-time correction matrix, combined with multi-frame super-resolution reconstruction technology, the image quality and detail performance during the switching process are significantly improved. By calculating the exposure compensation coefficient and color space conversion in real time, the exposure continuity and color consistency between different lenses are ensured, and the smoothness and image quality of dual-lens switching are improved.

[0041] In a specific embodiment, the process of executing step 101 may specifically include the following steps:

[0042] (1) Setting the initial voltage for the micro-electrically controlled liquid crystal optical element of the dual-lens motion camera to obtain the initial light refractive index distribution;

[0043] (2) Calculate the optical path of the incident light according to the refractive index distribution of the initial light to obtain the initial light transmission path, and perform optical simulation analysis on the initial light transmission path to obtain optical performance indicators;

[0044] (3) Iteratively optimize the voltage distribution of the micro-electrically controlled liquid crystal optical element according to the optical performance index to obtain a voltage control scheme, and generate a target optical path according to the voltage control scheme;

[0045] (4) Setting the exposure parameters of the image sensor of the dual-lens motion camera according to the target optical path to obtain an initial exposure configuration, and performing multi-scene sampling of the shooting environment based on the initial exposure configuration to obtain a sample video dataset;

[0046] (5) Perform image quality assessment on the sample video dataset to obtain an image quality score, and adaptively adjust the initial exposure configuration based on the image quality score to obtain an optimized exposure configuration;

[0047] (6) Based on the optimized exposure configuration and target optical path, video of multiple typical scenes is captured to obtain historical video data.

[0048] Specifically, the electrically controlled liquid crystal optical element adjusts the arrangement of liquid crystal molecules by applying voltage, thereby changing the refractive index of light. When setting the initial voltage, the voltage distribution Applied to the surface of liquid crystal optical elements, Indicates at location The voltage applied at each point. Based on the characteristics of the liquid crystal material, the refractive index at each point is calculated. , whose relationship can be expressed by a function Indicates that It is a function of the electro-optical properties of the liquid crystal material. The optical path of the incident light is calculated based on the principle of geometric optics. The purpose of the optical path calculation is to determine the path of the light after passing through the liquid crystal element. Light will refract in media with different refractive indices, and the refraction angle is given by Snell's law:

[0049] ;

[0050] in, and are the refractive indices of the medium into which the light enters and leaves it, respectively. and are the angle of incidence and the angle of refraction respectively. Based on these relationships, the initial light transmission path is calculated. After obtaining the initial light transmission path, optical simulation analysis is performed. Optical simulation analysis is used to evaluate the performance of optical systems, such as focal length, aberration, transmittance and other optical performance indicators. These indicators can be simulated and calculated by optical design software (such as Zemax or COMSOL). According to the optical performance indicators, the voltage distribution of the micro-electrically controlled liquid crystal optical element is iteratively optimized to find a voltage distribution that optimizes the performance of the optical system. The optimization process can be implemented using gradient descent method or genetic algorithm. Suppose the optimization objective function is , which is of the form:

[0051] ;

[0052] in, is the modulation transfer function, which represents the resolution capability of the system; is the depth of field, which indicates the focusing range of the system; Distortion is distortion, which indicates the geometric distortion of the image; is a weight coefficient, which indicates the importance of each performance index. After the voltage control scheme is obtained through iterative optimization, the target optical path is generated. This path reflects the propagation trajectory of light in the optical system under the optimized voltage distribution. The exposure parameters of the image sensor of the dual-lens motion camera are set according to the target optical path. The exposure parameters include aperture, shutter speed, and sensitivity (ISO). The initial exposure configuration can be determined by empirical values ​​or automatic exposure algorithms. In order to verify the effectiveness of the initial exposure configuration, multiple scenes of the shooting environment are sampled based on the initial exposure configuration to obtain a sample video dataset. These scenes cover different lighting conditions and dynamic ranges to comprehensively evaluate the system performance. When evaluating the image quality of the sample video dataset, some common image quality indicators such as signal-to-noise ratio (SNR), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) are used. The image quality score reflects the image quality in each scene by combining these indicators. The initial exposure configuration is adaptively adjusted according to the image quality score to obtain the optimized exposure configuration. The adaptive adjustment process can use machine learning algorithms such as random forests or support vector machines (SVM) to predict the optimal exposure parameters. Based on the optimized exposure configuration and target optical path, video capture of multiple typical scenes is performed to obtain historical video data. This data is used to further train and optimize the lens intelligent collaboration model, thereby improving the performance of the system in practical applications.

[0053] In a specific embodiment, the process of executing step 102 may specifically include the following steps:

[0054] (1) Perform spatiotemporal segmentation on historical video data to obtain multiple spatiotemporal blocks, and perform multi-scale convolutional neural network processing on each spatiotemporal block to obtain a primary feature map;

[0055] (2) Apply the attention mechanism to the primary feature map to obtain the weighted feature map, and perform non-local correlation analysis on the weighted feature map to obtain the global context feature;

[0056] (3) Cluster multiple spatiotemporal blocks according to global context features to obtain scene type labels, and model the temporal relationship of multiple spatiotemporal blocks according to the scene type labels to obtain a scene transition probability matrix;

[0057] (4) Construct a Markov decision process model based on the scene type label and the scene transition probability matrix to obtain the initial state transfer function, and perform Monte Carlo tree search optimization on the initial state transfer function to obtain the optimized state value function;

[0058] (5) A deep reinforcement learning network is constructed based on the optimized state-value function to obtain a lens coordination strategy network. The lens coordination strategy network is then subjected to multiple rounds of strategy iteration and experience replay training to obtain a lens intelligent coordination model.

[0059] Specifically, the video data is preprocessed. Video data is a three-dimensional data, two of which are spatial dimensions (width and height of the image) and the other dimension is the time dimension (frame sequence). Through spatiotemporal segmentation, the video data is sliced ​​according to the time dimension, and several frames in each time period constitute a spatiotemporal block. In the spatial dimension, each frame of the image is divided into multiple small blocks. Each spatiotemporal block consists of several small blocks of several consecutive frames. For each spatiotemporal block, a multi-scale convolutional neural network (CNN) is used for processing to extract features at different scales. The multi-scale CNN captures the detailed features and global features in the image by performing convolution operations on the image at different scales. After extracting the primary feature maps, the attention mechanism is applied to weight these feature maps. The attention mechanism can assign different weights to each part of the feature map according to its importance, thereby highlighting the key features. The calculation of the attention mechanism can be expressed by the formula:

[0060] Attention softmax ;

[0061] in, is the query matrix, is the key matrix, is the value matrix, is the dimension of the key vector. Through the attention mechanism, a weighted feature map is obtained. Non-local correlation analysis is performed on the weighted feature map to obtain global context features. Non-local correlation analysis captures the relationship between distant areas in the image by calculating the similarity between any two positions in the image, improves the ability of feature representation, and enables the model to better understand complex scenes. According to the global context features, multiple spatiotemporal blocks are clustered to obtain scene type labels. Cluster analysis can be achieved using the K-means algorithm by minimizing the distance between each spatiotemporal block and the center of the cluster to which it belongs. Scene type labels can help the system distinguish different scenes in video data and provide a basis for subsequent temporal relationship modeling. After obtaining the scene type labels, temporal relationship modeling is performed on multiple spatiotemporal blocks to construct a scene transition probability matrix. Temporal relationship modeling obtains the transition probability between scenes by analyzing the transition rules between different scenes. Let the scene transition probability matrix be ,in Indicates from the scene Switch to scene The probability of taking action Transfer to state The probability of. In order to optimize the state transition function, the Monte Carlo Tree Search (MCTS) method is used for optimization. MCTS finds the optimal state transition strategy by simulating multiple possible state transition paths. The optimized state value function can be expressed as:

[0062] ;

[0063] in, Yes Status The value of From the state Transfer to state Rewards, is a discount factor, which indicates the importance of future rewards. Based on the optimized state-value function, a deep reinforcement learning network is constructed. The deep reinforcement learning network obtains the lens coordination policy network by learning the state-value function. In order to improve the performance of the policy network, multiple rounds of policy iteration and experience replay training are performed on it. Policy iteration improves its decision-making ability under different states by repeatedly updating the policy network. Experience replay reduces the variance in the training process and improves training efficiency by storing and reusing previous training data. After multiple rounds of policy iteration and experience replay training, the lens intelligent coordination model is finally obtained. This model can automatically switch lenses according to the spatiotemporal characteristics and scene transition rules in the video data to provide the best shooting effect.

[0064] In a specific embodiment, the process of executing step 103 may specifically include the following steps:

[0065] (1) Obtain real-time video data from a dual-lens motion camera and perform parallel decoding on the real-time video data to obtain an original frame sequence;

[0066] (2) Perform optical flow estimation on the original frame sequence to obtain the motion vector field between adjacent frames, and perform temporal filtering on the original frame sequence based on the motion vector field to obtain a denoised frame sequence;

[0067] (3) Perform multi-scale pyramid decomposition on the denoised frame sequence to obtain multi-level image features, and construct a deep residual network based on the multi-level image features to obtain a frame-level semantic feature map;

[0068] (4) Perform self-attention calculation on the frame-level semantic feature map to obtain a key area heat map, and weight the frame-level semantic feature map according to the key area heat map to obtain an enhanced feature map;

[0069] (5) The enhanced feature map is processed by a graph convolutional network to obtain a local-global relationship matrix, and the importance score of each pixel is calculated based on the local-global relationship matrix to obtain a frame-level importance weight map.

[0070] Specifically, the video stream captured by the dual-lens motion camera is transmitted to the processing unit. Through an efficient video decoding algorithm, multiple video frames are processed in parallel to obtain the original frame sequence. Parallel decoding uses multi-threading technology to divide the video data into several parts, and each thread processes one part at the same time, thereby greatly improving the decoding speed. Optical flow estimation is performed on the original frame sequence. Optical flow estimation calculates the motion vector field by analyzing the pixel motion between adjacent frames. Assume and Respectively Frame and Frame at position The pixel value of the optical flow vector satisfy:

[0071] ;

[0072] By solving this equation, the motion vector field between adjacent frames is obtained. The optical flow estimation results are used to perform time domain filtering on the original frame sequence to obtain a denoised frame sequence. Time domain filtering reduces noise and artifacts by smoothing the motion between adjacent frames to obtain a clearer image sequence. Multi-scale pyramid decomposition is performed on the denoised frame sequence to extract multi-level image features. Multi-scale pyramid decomposition decomposes the image into multiple levels of different resolutions, each level contains information of different scales. In this way, global and local features in the image are captured. Multi-level image features are used to construct a deep residual network to obtain a frame-level semantic feature map. The deep residual network solves the gradient vanishing problem in deep network training by introducing residual connections and improves the representation ability of the network. Assume the input is , the output of the residual block is:

[0073] ;

[0074] in, represents the output after passing through several convolutional layers and nonlinear activation functions, is the weight parameter of the convolutional layer. The self-attention mechanism is calculated on the frame-level semantic feature map to generate a key area heat map. The self-attention mechanism highlights the key area by calculating the importance of each position in the feature map. The attention weight can be calculated by the following formula:

[0075] ;

[0076] in, Indicates location and location The similarity between them. The frame-level semantic feature map is weighted by attention weight to obtain the enhanced feature map. The enhanced feature map is processed by graph convolution network (GCN) to obtain the local-global relationship matrix. The feature representation of the layer is:

[0077] ;

[0078] in, For the The weight matrix of the layer, is a nonlinear activation function. According to the local-global relationship matrix, the importance score of each pixel is calculated to obtain the frame-level importance weight map. The importance score reflects the importance of each pixel in the entire image and helps identify the key areas in the image.

[0079] In a specific embodiment, the process of executing step 104 may specifically include the following steps:

[0080] (1) Perform a three-dimensional convolution operation on the frame-level importance weight map to obtain a spatiotemporal feature tensor, and then weight the spatiotemporal feature tensor to obtain a weighted spatiotemporal feature;

[0081] (2) Perform autoencoder compression based on weighted spatiotemporal features to obtain compressed feature representation, and construct a variational inference network based on the compressed feature representation to obtain feature distribution parameters;

[0082] (3) The feature distribution parameters are sampled through the lens intelligent collaborative model to obtain multiple groups of feature samples, and a Gaussian process regression model is trained based on the multiple groups of feature samples to obtain a probability prediction function;

[0083] (4) The probability prediction function is optimized by Bayesian optimization to obtain the optimal hyperparameters. Sequence rolling prediction and curve construction are performed based on the optimal hyperparameters to obtain the shot switching probability prediction curve.

[0084] Specifically, a three-dimensional convolution operation is performed on the frame-level importance weight map to obtain a spatiotemporal feature tensor. The three-dimensional convolutional neural network performs convolution operations not only in the spatial dimension, but also in the temporal dimension, thereby capturing the spatiotemporal features in the video data. Suppose the input weight map is , the three-dimensional convolution kernel is , then the convolution operation can be expressed as:

[0085] ;

[0086] in, represents the position in three-dimensional space, is the radius of the convolution kernel in each dimension. Through the convolution operation, we get the spatiotemporal feature tensor The spatiotemporal feature tensor is weighted to highlight important features. The weighting process can be achieved through the attention mechanism. Let the attention weight be , then the weighted spatiotemporal features It can be expressed as:

[0087] ;

[0088] Attention Weight The self-attention calculation of the spatiotemporal feature tensor can enhance the feature representation of the key area. The autoencoder is compressed according to the weighted spatiotemporal features. The autoencoder is an unsupervised learning model that compresses high-dimensional input data into a low-dimensional space through an encoder and then restores it through a decoder. Let the encoder function be , the decoder function is , the input data is , then the goal of the autoencoder is to minimize the reconstruction error:

[0089] ;

[0090] Through autoencoder compression, compressed feature representation is obtained , reducing the feature dimension and retaining important information. , construct a variational inference network and obtain the characteristic distribution parameters. The variational inference network introduces random variables , model the feature distribution. is the true posterior distribution, For a variational distribution, the goal of variational inference is to minimize the KL divergence between the two:

[0091] ;

[0092] By optimizing the variational distribution, we get the characteristic distribution parameters and ,in represents the mean, Denotes the standard deviation. Through the lens intelligent collaborative model, the feature distribution parameters are sampled to obtain multiple sets of feature samples. Let the feature sample be , by repeated sampling, multiple samples are generated These feature samples are used to train the Gaussian process regression model. Gaussian process regression is a non-parametric Bayesian regression method that predicts the probability distribution of new data points by modeling sample data. Suppose the training sample is ,in are input features, is the output, then the prediction distribution of the Gaussian process is:

[0093] ;

[0094] in, is the kernel function, and are the input and output matrices of the training samples, respectively. Bayesian optimization is performed on the probability prediction function to find the optimal hyperparameters. Bayesian optimization finds the optimal hyperparameter combination by maximizing the posterior probability of the objective function. Let the objective function be , the hyperparameters are , then the goal of Bayesian optimization is:

[0095] ;

[0096] Where D is the existing data. Through Bayesian optimization, the optimal hyperparameters are obtained. , and perform sequence rolling prediction and curve construction based on these hyperparameters, and finally obtain the shot switching probability prediction curve.

[0097] In a specific embodiment, the process of executing step 105 may specifically include the following steps:

[0098] (1) Perform wavelet transform on the shot switching probability prediction curve to obtain multi-scale frequency features, and calculate the dynamic threshold based on the multi-scale frequency features to obtain the initial switching time set;

[0099] (2) Perform cluster analysis on the initial switching time set to obtain candidate switching intervals, and construct a state transfer matrix based on the candidate switching intervals;

[0100] (3) Generate an initial policy network based on the state transition matrix, and design a reward function based on the initial policy network to obtain a reinforcement learning environment;

[0101] (4) Perform proximal strategy optimization on the reinforcement learning environment to obtain an optimized strategy network, and generate multiple sets of switching strategies based on the optimized strategy network to obtain a strategy sample set;

[0102] (5) Cross-validate the strategy sample set to obtain performance evaluation indicators, and sort and screen the strategy sample set according to the performance evaluation indicators to obtain the optimal lens switching control strategy.

[0103] Specifically, the prediction curve is subjected to discrete wavelet transform. Wavelet transform can decompose the signal into different frequency components and reveal the characteristics of the signal at multiple scales. Suppose the prediction curve is , then the discrete wavelet transform can be expressed as:

[0104] ;

[0105] in, Indicated in scale and location The wavelet coefficients of is the wavelet basis function. By calculating the wavelet coefficients, the multi-scale frequency characteristics are obtained. According to the multi-scale frequency characteristics, the dynamic threshold is calculated to identify the initial switching time. The dynamic threshold is based on the change of frequency characteristics. Let the threshold function be , then the initial switching time set can be expressed as the time that satisfies the following conditions:

[0106] ;

[0107] Perform cluster analysis on the initial switching time set. Cluster analysis can use the K-means algorithm to minimize the distance between the time points and cluster similar switching time points together to form candidate switching intervals. Suppose the initial switching time set is , then the clustering results can be expressed as several clusters , each cluster represents a candidate switching interval. According to the candidate switching intervals, a state transfer matrix is ​​constructed. The state transfer matrix describes the transition probability between different states. Let the state transfer matrix be ,in Indicates from the state Transfer to state The probability of state transition is calculated by analyzing the switching moments in the candidate switching interval. The initial policy network is generated based on the state transition matrix. The initial policy network predicts the next state through the state transition matrix and takes action based on the transition probability. Designing a reward function is a key step in reinforcement learning. Indicates in status Take Action The reward function can be designed according to the effect of the switching strategy. The reinforcement learning environment is constructed through the initial policy network and the reward function. The reinforcement learning environment simulates the lens switching process and provides feedback for training the policy network. Proximal Policy Optimization (PPO) is used for training. PPO improves the stability of training by limiting the amplitude of policy updates. Suppose the old policy is , the new strategy is , then the objective function of PPO can be expressed as:

[0108] ;

[0109] in, is the advantage function, is a hyperparameter that controls the magnitude of the update. Through PPO training, an optimized policy network is obtained. Based on the optimized policy network, multiple sets of switching strategies are generated to form a policy sample set. The policy sample set contains lens switching strategies in different scenarios for evaluation and screening. The policy sample set is cross-validated, and the performance evaluation indicators are obtained by testing the performance of each strategy on different data sets. Performance evaluation indicators include accuracy, stability, and switching smoothness. According to the performance evaluation indicators, the policy sample set is sorted and screened, and finally the best lens switching control strategy is obtained. The best strategy can achieve efficient lens switching in different scenarios and provide the best video quality and user experience.

[0110] In a specific embodiment, the process of executing step 106 may specifically include the following steps:

[0111] (1) According to the optimal lens switching control strategy, the adjustment sequence of the target optical path is calculated to obtain a segmented adjustment scheme, and Bezier curve interpolation is performed on the segmented adjustment scheme to obtain a smooth transition path;

[0112] (2) Voltage modulation of the micro-electrically controlled liquid crystal optical element is performed according to the smooth transition path to obtain a dynamic refractive index distribution;

[0113] (3) Perform ray tracing simulation on the dynamic refractive index distribution to obtain the instantaneous optical characteristic curve, and calculate the image distortion compensation parameters based on the instantaneous optical characteristic curve to obtain the real-time correction matrix;

[0114] (4) Perform bilinear interpolation on the real-time correction matrix to obtain a pixel-level mapping relationship, and perform geometric transformation based on the pixel-level mapping relationship to obtain a corrected image sequence;

[0115] (5) Perform multi-frame super-resolution reconstruction on the corrected image sequence to obtain a high-resolution image sequence, and calculate the exposure compensation coefficient and color space conversion based on the high-resolution image sequence to obtain the target video data.

[0116] Specifically, the adjustment sequence of the target optical path is calculated according to the optimal lens switching control strategy. The optimal lens switching control strategy provides the optical path adjustment information required at different time points. These adjustment information are processed in segments to form a segmented adjustment scheme. In order to ensure the smooth transition of the optical path adjustment, Bezier curve interpolation is applied to the segmented adjustment scheme. The Bezier curve defines the smooth transition of the path through control points, and the interpolation function can be expressed as:

[0117] ;

[0118] in, Represents the Bezier curve in time location, is the control point, is the binomial coefficient. Through Bezier curve interpolation, a smooth transition path is obtained. According to the smooth transition path, the voltage of the micro-electrically controlled liquid crystal optical element is modulated to obtain a dynamic refractive index distribution. The purpose of voltage modulation is to change the refractive index distribution by adjusting the arrangement state of liquid crystal molecules. Let the voltage distribution function be , the refractive index distribution function is , then the relationship between the two can be obtained by fitting the experimental data, usually expressed as:

[0119] ;

[0120] in, is the electro-optical property function of the liquid crystal material. Ray tracing simulation is performed on the dynamic refractive index distribution to calculate the instantaneous optical characteristic curve. Ray tracing calculates the performance index of the optical system by simulating the propagation path of light in media with different refractive indices. Assume that the incident angle of the light is , the refraction angle is :

[0121] ;

[0122] Through ray tracing, the instantaneous optical characteristic curve is obtained. According to the instantaneous optical characteristic curve, the image distortion compensation parameters are calculated and the real-time correction matrix is ​​obtained. The image distortion compensation parameters are used to correct the image distortion caused by optical elements. Assume that the distortion model is , the correction matrix is , then the corrected pixel position can be expressed as:

[0123] ;

[0124] In order to improve the correction accuracy, bilinear interpolation is performed on the real-time correction matrix to obtain the pixel-level mapping relationship. Bilinear interpolation calculates the position of the target pixel by weighted averaging the four adjacent pixels of the image. The interpolation formula is:

[0125] ;

[0126] in, represents the pixel value, are the adjacent pixel positions, and is the interpolation coefficient. According to the pixel-level mapping relationship, the image is geometrically transformed to obtain a corrected image sequence. Geometric transformation corrects the distortion in the image by adjusting the position and shape of the pixels. After obtaining the corrected image sequence, multi-frame super-resolution reconstruction is performed on it to generate a high-resolution image sequence. Super-resolution reconstruction generates a high-resolution image by fusing the information of multiple low-resolution images. According to the high-resolution image sequence, the exposure compensation coefficient and color space conversion are calculated to obtain the target video data. Exposure compensation ensures the clarity of the image in different scenes by adjusting the brightness of the image. Let the exposure compensation coefficient be , then the compensated image can be expressed as:

[0127] ;

[0128] Color space conversion ensures the color consistency of an image by converting it from one color space to another. Common conversions include conversion from RGB to YUV, and the conversion formula is:

[0129] ;

[0130] The above describes the switching method of the dual-lens motion camera in the embodiment of the present application. The following describes the switching device of the dual-lens motion camera in the embodiment of the present application. Figure 2 In one embodiment of the present application, a switching device for a dual-lens motion camera includes:

[0131] Configuration module 201, used to configure the light transmission path of the dual-lens motion camera, obtain the target optical path and collect historical video data;

[0132] An extraction module 202 is used to extract scene features from historical video data, obtain scene feature data and build a shot intelligent collaboration model;

[0133] The analysis module 203 is used to obtain real-time video data and perform real-time analysis of the video sequence frame by frame to obtain a frame-level importance weight map;

[0134] A modeling module 204 is used to perform multi-dimensional feature extraction and time series modeling on the frame-level importance weight map through a lens intelligent collaborative model to obtain a lens switching probability prediction curve;

[0135] A judgment module 205 is used to perform threshold judgment and reinforcement learning analysis on the shot switching probability prediction curve to obtain an optimal shot switching control strategy;

[0136] The adjustment module 206 is used to progressively adjust the target optical path through the optimal lens switching control strategy and generate target video data.

[0137] Through the collaboration of the above components and the precise control of micro-electrically controlled liquid crystal optical elements, the real-time dynamic adjustment of the light transmission path is realized, which greatly improves the flexibility and response speed of the optical system and effectively reduces the delay and error caused by traditional mechanical switching. The constructed lens intelligent collaborative model can analyze and predict complex scenes in real time, accurately grasp the best switching time, and improve the degree of automation and shooting effect. The deep feature extraction and temporal relationship modeling of video data improve the accuracy of switching probability prediction. Combined with dynamic threshold judgment and reinforcement learning analysis, the adaptive optimization of the switching strategy is realized, and the switching behavior can be automatically adjusted according to different scenes and user habits, which improves the intelligence and user experience of the system. Through Bezier curve interpolation and real-time ray tracing technology, the smooth transition of the optical path is achieved, which effectively reduces the jitter and distortion of the picture during the switching process. By using dynamic refractive index distribution calculation and real-time correction matrix, combined with multi-frame super-resolution reconstruction technology, the image quality and detail performance during the switching process are significantly improved. By calculating the exposure compensation coefficient and color space conversion in real time, the exposure continuity and color consistency between different lenses are ensured, and the smoothness and image quality of dual-lens switching are improved.

[0138] The present application also provides a computer device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the dual-lens motion camera switching method in the above-mentioned embodiments.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0141] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for switching a dual-lens motion camera, characterized in that: The method comprises: The light transmission path of the dual-lens motion camera is configured to obtain a target optical path and collect historical video data; specifically, the method includes: setting an initial voltage for a micro-electrically controlled liquid crystal optical element of the dual-lens motion camera to obtain an initial light refractive index distribution; calculating the optical path of the incident light according to the initial light refractive index distribution to obtain an initial light transmission path, and performing optical simulation analysis on the initial light transmission path to obtain an optical performance index; iteratively optimizing the voltage distribution of the micro-electrically controlled liquid crystal optical element according to the optical performance index to obtain a voltage control scheme, and generating a target optical path according to the voltage control scheme; setting exposure parameters for the image sensor of the dual-lens motion camera according to the target optical path to obtain an initial exposure configuration, and performing multi-scene sampling of the shooting environment based on the initial exposure configuration to obtain a sample video data set; performing image quality evaluation on the sample video data set to obtain an image quality score, and adaptively adjusting the initial exposure configuration according to the image quality score to obtain an optimized exposure configuration; based on the optimized exposure configuration and the target optical path, performing video acquisition on multiple typical scenes to obtain historical video data; Extract scene features from the historical video data to obtain scene feature data and construct a lens intelligent collaboration model; specifically comprising: performing spatiotemporal segmentation on the historical video data to obtain multiple spatiotemporal blocks, and performing multi-scale convolutional neural network processing on each spatiotemporal block to obtain a primary feature map; applying an attention mechanism to the primary feature map to obtain a weighted feature map, and performing non-local correlation analysis on the weighted feature map to obtain global context features; clustering the multiple spatiotemporal blocks according to the global context features to obtain scene type labels, and performing temporal relationship modeling on the multiple spatiotemporal blocks according to the scene type labels to obtain a scene transition probability matrix; constructing a Markov decision process model according to the scene type labels and the scene transition probability matrix to obtain an initial state transfer function, and performing Monte Carlo tree search optimization on the initial state transfer function to obtain an optimized state value function; constructing a deep reinforcement learning network according to the optimized state value function to obtain a lens collaboration strategy network, and performing multiple rounds of strategy iteration and experience playback training on the lens collaboration strategy network to obtain a lens intelligent collaboration model; Acquire real-time video data and perform real-time analysis of the video sequence frame by frame to obtain a frame-level importance weight map; specifically, the method includes: acquiring real-time video data of the dual-lens motion camera, and performing parallel decoding processing on the real-time video data to obtain an original frame sequence; performing optical flow estimation on the original frame sequence to obtain a motion vector field between adjacent frames, and performing time domain filtering on the original frame sequence according to the motion vector field to obtain a denoised frame sequence; performing multi-scale pyramid decomposition on the denoised frame sequence to obtain multi-level image features, and constructing a deep residual network based on the multi-level image features to obtain a frame-level semantic feature map; performing self-attention mechanism calculation on the frame-level semantic feature map to obtain a key area heat map, and weighting the frame-level semantic feature map according to the key area heat map to obtain an enhanced feature map; performing graph convolution network processing on the enhanced feature map to obtain a local-global relationship matrix, and calculating the importance score of each pixel according to the local-global relationship matrix to obtain a frame-level importance weight map; The frame-level importance weight map is subjected to multi-dimensional feature extraction and time series modeling through the lens intelligent collaborative model to obtain a lens switching probability prediction curve; specifically comprising: performing a three-dimensional convolution operation on the frame-level importance weight map to obtain a spatiotemporal feature tensor, and weighting the spatiotemporal feature tensor to obtain a weighted spatiotemporal feature; performing autoencoder compression according to the weighted spatiotemporal feature to obtain a compressed feature representation, and constructing a variational inference network according to the compressed feature representation to obtain feature distribution parameters; sampling the feature distribution parameters through the lens intelligent collaborative model to obtain multiple groups of feature samples, and training a Gaussian process regression model according to the multiple groups of feature samples to obtain a probability prediction function; performing Bayesian optimization on the probability prediction function to obtain an optimal hyperparameter, and performing sequence rolling prediction and curve construction according to the optimal hyperparameter to obtain a lens switching probability prediction curve; The lens switching probability prediction curve is subjected to threshold judgment and reinforcement learning analysis to obtain an optimal lens switching control strategy; specifically, the method comprises: performing wavelet transform on the lens switching probability prediction curve to obtain multi-scale frequency features, and calculating dynamic thresholds according to the multi-scale frequency features to obtain an initial switching moment set; performing cluster analysis on the initial switching moment set to obtain candidate switching intervals, and constructing a state transfer matrix according to the candidate switching intervals; generating an initial policy network according to the state transfer matrix, and designing a reward function according to the initial policy network to obtain a reinforcement learning environment; performing proximal policy optimization on the reinforcement learning environment to obtain an optimized policy network, and generating multiple groups of switching strategies according to the optimized policy network to obtain a policy sample set; performing cross-validation on the policy sample set to obtain a performance evaluation index, and sorting and screening the policy sample set according to the performance evaluation index to obtain an optimal lens switching control strategy; The target optical path is progressively adjusted and target video data is generated by the optimal lens switching control strategy; specifically, the method includes: calculating the adjustment sequence of the target optical path according to the optimal lens switching control strategy to obtain a segmented adjustment scheme, and performing Bezier curve interpolation on the segmented adjustment scheme to obtain a smooth transition path; performing voltage modulation on the micro-electrically controlled liquid crystal optical element according to the smooth transition path to obtain a dynamic refractive index distribution; performing ray tracing simulation on the dynamic refractive index distribution to obtain an instantaneous optical characteristic curve, and calculating image distortion compensation parameters according to the instantaneous optical characteristic curve to obtain a real-time correction matrix; performing bilinear interpolation on the real-time correction matrix to obtain a pixel-level mapping relationship, and performing geometric transformation according to the pixel-level mapping relationship to obtain a corrected image sequence; performing multi-frame super-resolution reconstruction on the corrected image sequence to obtain a high-resolution image sequence, and calculating an exposure compensation coefficient and color space conversion according to the high-resolution image sequence to obtain target video data.

2. A switching device for a dual-lens motion camera, characterized in that: The device comprises: A configuration module is used to configure the light transmission path of a dual-lens motion camera, obtain a target optical path and collect historical video data; specifically comprising: setting an initial voltage for a micro-electrically controlled liquid crystal optical element of the dual-lens motion camera to obtain an initial light refractive index distribution; calculating the optical path of the incident light according to the initial light refractive index distribution to obtain an initial light transmission path, and performing optical simulation analysis on the initial light transmission path to obtain an optical performance index; iteratively optimizing the voltage distribution of the micro-electrically controlled liquid crystal optical element according to the optical performance index to obtain a voltage control scheme, and generating a target optical path according to the voltage control scheme; setting exposure parameters for the image sensor of the dual-lens motion camera according to the target optical path to obtain an initial exposure configuration, and performing multi-scene sampling of the shooting environment based on the initial exposure configuration to obtain a sample video data set; performing image quality evaluation on the sample video data set to obtain an image quality score, and adaptively adjusting the initial exposure configuration according to the image quality score to obtain an optimized exposure configuration; based on the optimized exposure configuration and the target optical path, performing video acquisition on multiple typical scenes to obtain historical video data; The extraction module is used to extract scene features from the historical video data, obtain scene feature data and construct a lens intelligent collaboration model; specifically comprising: performing spatiotemporal segmentation on the historical video data to obtain multiple spatiotemporal blocks, and performing multi-scale convolutional neural network processing on each spatiotemporal block to obtain a primary feature map; applying an attention mechanism to the primary feature map to obtain a weighted feature map, and performing non-local correlation analysis on the weighted feature map to obtain a global context feature; clustering the multiple spatiotemporal blocks according to the global context feature to obtain a scene type label, and performing temporal relationship modeling on the multiple spatiotemporal blocks according to the scene type label to obtain a scene transition probability matrix; constructing a Markov decision process model according to the scene type label and the scene transition probability matrix to obtain an initial state transfer function, and performing Monte Carlo tree search optimization on the initial state transfer function to obtain an optimized state value function; constructing a deep reinforcement learning network according to the optimized state value function to obtain a lens collaboration strategy network, and performing multiple rounds of strategy iteration and experience playback training on the lens collaboration strategy network to obtain a lens intelligent collaboration model; The analysis module is used to obtain real-time video data and perform real-time frame-by-frame analysis of the video sequence to obtain a frame-level importance weight map; specifically, it includes: obtaining real-time video data of the dual-lens motion camera, and performing parallel decoding processing on the real-time video data to obtain an original frame sequence; performing optical flow estimation on the original frame sequence to obtain a motion vector field between adjacent frames, and performing time domain filtering on the original frame sequence according to the motion vector field to obtain a denoised frame sequence; performing multi-scale pyramid decomposition on the denoised frame sequence to obtain multi-level image features, and constructing a deep residual network based on the multi-level image features to obtain a frame-level semantic feature map; performing self-attention mechanism calculation on the frame-level semantic feature map to obtain a key area heat map, and weighting the frame-level semantic feature map according to the key area heat map to obtain an enhanced feature map; performing graph convolution network processing on the enhanced feature map to obtain a local-global relationship matrix, and calculating the importance score of each pixel according to the local-global relationship matrix to obtain a frame-level importance weight map; A modeling module is used to perform multi-dimensional feature extraction and time series modeling on the frame-level importance weight map through the lens intelligent collaborative model to obtain a lens switching probability prediction curve; specifically comprising: performing a three-dimensional convolution operation on the frame-level importance weight map to obtain a spatiotemporal feature tensor, and weighting the spatiotemporal feature tensor to obtain a weighted spatiotemporal feature; performing autoencoder compression according to the weighted spatiotemporal feature to obtain a compressed feature representation, and constructing a variational inference network according to the compressed feature representation to obtain feature distribution parameters; sampling the feature distribution parameters through the lens intelligent collaborative model to obtain multiple groups of feature samples, and training a Gaussian process regression model according to the multiple groups of feature samples to obtain a probability prediction function; performing Bayesian optimization on the probability prediction function to obtain an optimal hyperparameter, and performing sequence rolling prediction and curve construction according to the optimal hyperparameter to obtain a lens switching probability prediction curve; A judgment module is used to perform threshold judgment and reinforcement learning analysis on the lens switching probability prediction curve to obtain an optimal lens switching control strategy; specifically comprising: performing wavelet transform on the lens switching probability prediction curve to obtain multi-scale frequency characteristics, and calculating dynamic thresholds based on the multi-scale frequency characteristics to obtain an initial switching moment set; performing cluster analysis on the initial switching moment set to obtain candidate switching intervals, and constructing a state transfer matrix based on the candidate switching intervals; generating an initial policy network based on the state transfer matrix, and designing a reward function based on the initial policy network to obtain a reinforcement learning environment; performing proximal policy optimization on the reinforcement learning environment to obtain an optimized policy network, and generating multiple groups of switching strategies based on the optimized policy network to obtain a policy sample set; performing cross-validation on the policy sample set to obtain a performance evaluation index, and sorting and screening the policy sample set based on the performance evaluation index to obtain an optimal lens switching control strategy; The adjustment module is used to progressively adjust the target optical path through the optimal lens switching control strategy and generate target video data; specifically comprising: calculating the adjustment sequence of the target optical path according to the optimal lens switching control strategy to obtain a segmented adjustment scheme, and performing Bezier curve interpolation on the segmented adjustment scheme to obtain a smooth transition path; performing voltage modulation on the micro-electrically controlled liquid crystal optical element according to the smooth transition path to obtain a dynamic refractive index distribution; performing ray tracing simulation on the dynamic refractive index distribution to obtain an instantaneous optical characteristic curve, and calculating image distortion compensation parameters according to the instantaneous optical characteristic curve to obtain a real-time correction matrix; performing bilinear interpolation on the real-time correction matrix to obtain a pixel-level mapping relationship, and performing geometric transformation according to the pixel-level mapping relationship to obtain a corrected image sequence; performing multi-frame super-resolution reconstruction on the corrected image sequence to obtain a high-resolution image sequence, and calculating exposure compensation coefficients and color space conversion according to the high-resolution image sequence to obtain target video data.

3. A computer device, characterized in that: The computer device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the computer device to execute the switching method of the dual-lens motion camera as claimed in claim 1.

Citation Information

Patent Citations

  • Automatic switching method and device for double cameras of video conference

    CN111866437A

  • Multi-lens switching display method and system, intelligent terminal and storage medium

    CN112637573A

  • Football shooting lens switching method and system, computer equipment and storage medium

    CN114928721A

  • Ultra-high-definition on-site rebroadcasting and live broadcasting camera system

    CN118200484A