Motion detection method and device and electronic equipment

Abstract

The invention discloses a motion detection method and device and electronic equipment, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a left pixel sub-image and a right pixel sub-image of a current shooting scene acquired by a dual-pixel sensor of a camera; inputting the left pixel sub-image and the right pixel sub-image into a first network, and performing feature extraction on the left pixel sub-image and the right pixel sub-image to obtain a depth image and a disparity image corresponding to the current shooting scene; inputting the fused image, the depth map and the disparity map into a second network, analyzing the fused image, the depth map and the disparity map, and outputting an optical flow graph of the current shooting scene; the fused image is obtained by fusing the left pixel sub-image and the right pixel sub-image; and determining motion information of the moving object in the current shooting scene according to the optical flow graph.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a motion detection method and device and electronic equipment. BACKGROUND

[0002] As a core technology of computer vision and video analysis, motion detection realizes key applications in mobile phone motion snapshot, image anti-shake, main body focusing and other scenes by identifying the changed area between continuous images.

[0003] Current mainstream algorithms include inter-frame difference method and optical flow method. The inter-frame difference method extracts the motion area through adjacent frame difference operation, but can only detect the pixel change area and cannot obtain target motion direction, speed and other vector information, and is prone to false detection due to environmental interference. The optical flow method estimates motion vectors based on pixel space-time changes, but its performance is highly dependent on light stability, and sudden changes in light will cause distortion in optical flow field calculation. When an object is occluded or moves in a large scale, abnormal motion vectors are prone to be generated.

[0004] Therefore, the current motion detection scheme has insufficient detection accuracy in complex scenes. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a motion detection method, device and electronic equipment, which can improve the accuracy of motion detection.

[0006] In a first aspect, the embodiments of the present application provide a motion detection method, which comprises:

[0007] obtaining a left pixel subgraph and a right pixel subgraph of a current shooting scene collected by a dual-pixel sensor of a camera;

[0008] inputting the left pixel subgraph and the right pixel subgraph into a first network, performing feature extraction on the left pixel subgraph and the right pixel subgraph, and obtaining a depth map and a disparity map corresponding to the current shooting scene;

[0009] inputting a fusion image, the depth map and the disparity map into a second network, analyzing the fusion image, the depth map and the disparity map, and outputting an optical flow map of the current shooting scene; the fusion image is obtained by fusing the left pixel subgraph and the right pixel subgraph;

[0010] determining motion information of a moving object in the current shooting scene according to the optical flow map.

[0011] In a second aspect, the embodiments of the present application provide a motion detection device, which comprises:

[0012] an acquisition module configured to acquire a left pixel subgraph and a right pixel subgraph of a current shooting scene collected by a dual-pixel sensor of a camera;

[0013] a first input module, configured to input the left pixel subgraph and the right pixel subgraph into a first network, and perform feature extraction on the left pixel subgraph and the right pixel subgraph to obtain a depth map and a disparity map corresponding to the current shooting scene;

[0014] a second input module, configured to input a fusion image, the depth map and the disparity map into a second network, and perform analysis on the fusion image, the depth map and the disparity map to output an optical flow map of the current shooting scene; the fusion image is obtained by fusing the left pixel subgraph and the right pixel subgraph;

[0015] a detection module, configured to determine motion information of a moving object in the current shooting scene according to the optical flow map.

[0016] In a third aspect, an electronic device is provided, which includes a processor and a memory. The memory stores programs or instructions executable on the processor. When the programs or instructions are executed by the processor, the steps of the method according to the first aspect are implemented.

[0017] In a fourth aspect, a readable storage medium is provided. The readable storage medium stores programs or instructions. When the programs or instructions are executed by a processor, the steps of the method according to the first aspect are implemented.

[0018] In a fifth aspect, a chip is provided. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is configured to execute programs or instructions to implement the method according to the first aspect.

[0019] In a sixth aspect, a computer program product is provided. The program product is stored in a storage medium. The program product is executed by at least one processor to implement the method according to the first aspect.

[0020] In the embodiments of the present application, by obtaining a left pixel subgraph and a right pixel subgraph of a current shooting scene collected by a dual-pixel sensor of a camera, the left pixel subgraph and the right pixel subgraph contain spatial angle difference information, which provides data support for solving motion detection in an out-of-focus scene. The left pixel subgraph and the right pixel subgraph are input into a first network, and feature extraction is performed on the left pixel subgraph and the right pixel subgraph. The phase difference characteristics of the dual-pixel subgraph are used. Even if the target is blurred, a reliable depth map and disparity map can still be generated, which provides key spatial geometric constraints for subsequent motion analysis.

[0021] The fused image, depth map, and disparity map are input into the second network. The fused image is obtained by fusing the left and right pixel sub-images, preserving the complete brightness and texture information of the two pixels. The second network is a deep learning model containing a temporal recurrent structure, which can jointly analyze the fused image, depth map, and disparity map of the current frame using historical frame information. It outputs an optical flow map to characterize the motion direction and velocity vector of pixels between consecutive frames, improving the robustness of optical flow estimation in complex scenes. Based on the optical flow map, the motion information of moving objects in the current shooting scene can be detected, which can improve the accuracy of motion detection. Attached Figure Description

[0022] Figure 1 This is a flowchart of a motion detection method provided in an embodiment of this application;

[0023] Figure 2 This is a schematic diagram of a left pixel sub-image and a right pixel sub-image provided in an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of an object's motion provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of a first network provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of a second network provided in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of an optical flow graph provided in an embodiment of this application;

[0028] Figure 7 This is a structural diagram of a motion detection device provided in an embodiment of this application;

[0029] Figure 8 This is one of the hardware structure diagrams of the electronic device according to an embodiment of this application;

[0030] Figure 9 This is the second schematic diagram of the hardware structure of the electronic device according to an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0032] The terms "first," "second," etc., used in this specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0033] The motion detection method provided in this application embodiment can be applied to at least the following application scenarios, which will be described below.

[0034] Currently, in the field of image capture, motion detection technology and image signal processors work together to achieve intelligent capture of moving targets. For example, in high-speed motion scenes, motion detection algorithms based on optical flow can calculate the trajectory of objects in real time, drive the phase focusing module to dynamically adjust the focal length, and work with dual-pixel sensors to achieve millisecond-level tracking focus response, ensuring that the moving subject remains clear even under high-speed displacement.

[0035] In video stabilization, global motion vectors are detected using the inter-frame difference method, and combined with gyroscope inertial data, pixel-level compensation is performed to compensate for image shifts caused by hand tremors. This achieves a fusion optimization of electronic and optical image stabilization, improving image stability in complex motion scenes. At the interactive experience level, motion detection technology is becoming a key support for natural human-computer interaction. Deep learning-based optical flow estimation models can accurately analyze the motion trajectory of user gestures, enabling contactless interaction functions such as air gesture operations and dynamic password input.

[0036] In view of the problems in the related technologies, the embodiments of this application provide a motion detection method, device and electronic device, which can solve the problem of low accuracy of motion detection in the related technologies.

[0037] The motion detection method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0038] Figure 1 This is a flowchart of a motion detection method provided in an embodiment of this application.

[0039] like Figure 1 As shown, the motion detection method may include steps 110-140. This method is applied to a motion detection device, as detailed below:

[0040] Step 110: Obtain the left and right pixel sub-images of the current shooting scene captured by the dual-pixel sensor of the camera;

[0041] A dual-pixel sensor divides each imaging pixel into left and right sub-pixels. The left and right sub-images are acquired by two separate sub-pixel arrays, and the difference in the angle of light received by the two sub-images creates phase difference information similar to binocular vision. Dual-pixel technology integrates two independent photodiodes within each pixel of the sensor to capture phase difference information separately, thus achieving fast focusing. The left and right sub-images carry the phase difference information of the scene, and this phase difference characteristic based on the hardware structure can be stably preserved even if the target is in an out-of-focus, blurred state.

[0042] like Figure 2 As shown, dual-pixel data includes a left pixel sub-image and a right pixel sub-image. When the target image is blurred, there is a phase difference between the left and right pixel sub-images; when the target image is sharp, the phase difference disappears. This phase difference characteristic can be directly used to drive focusing operations. When the illumination changes, the amount of light signal received by the left and right pixel sub-images changes synchronously, so their phase difference remains stable and is not affected by illumination fluctuations. At the same time, depth information can help the model analyze real three-dimensional spatial changes, avoiding scene misjudgments caused by relying solely on pixel changes.

[0043] In a shooting scenario where an athlete is running, if the focus point is not on the athlete or has not yet achieved focus, the left and right sub-images of the athlete's area will have a phase difference due to differences in the imaging optical path, resulting in their pixel positions not completely coinciding and causing the athlete to appear blurry in the image. However, if the focus point accurately falls on the athlete and achieves focus, the left and right sub-images of the athlete's area will completely overlap because the light converges on the same focal plane, at which point the phase difference will be zero, and the athlete will be clearly imaged. To obtain the left and right sub-image data of the current shooting scene, such as... Figure 3 As shown, the athlete only has horizontal displacement, and its trajectory does not involve vertical displacement along the optical axis of the camera.

[0044] Therefore, by acquiring the left and right pixel sub-images of the current shooting scene collected by the camera's dual-pixel sensor, a raw data foundation is provided for subsequent processing. Compared with the limitation of traditional single-pixel sensors that can only acquire a single image, the spatial angle difference information contained in the dual-pixel sub-image provides hardware-level data support for solving the motion detection problem in out-of-focus scenes.

[0045] Step 120: Input the left pixel sub-image and the right pixel sub-image into the first network, extract features from the left pixel sub-image and the right pixel sub-image to obtain the depth map and disparity map corresponding to the current shooting scene;

[0046] The first network is a convolutional neural network optimized for dual-pixel data. It constructs depth maps and disparity maps for each pixel in the scene by extracting the phase difference features and texture information of the left and right pixel sub-images. The depth map is used to represent the distance from the pixel to the camera, and the disparity map is used to represent the positional deviation of corresponding pixels in the left and right pixel sub-images.

[0047] The first network optimizes the convolutional layer and attention mechanism for the phase difference features between the left and right pixel sub-images. By extracting features such as pixel position deviation and texture correlation between sub-images, and combining geometric constraints, it calculates the depth map and disparity map.

[0048] Therefore, by extracting features from the left and right pixel sub-images, the depth map and disparity map of the current shooting scene are obtained. This breaks through the limitation of traditional methods that rely on clear textures for depth estimation. It can utilize the phase difference characteristics of the two pixel sub-images to generate reliable depth and disparity information even if the target is blurred, providing key spatial geometric constraints for subsequent motion analysis.

[0049] Step 130: Input the fused image, the depth map, and the disparity map into the second network, analyze the fused image, the depth map, and the disparity map, and output the optical flow map of the current shooting scene; the fused image is obtained by fusing the left pixel sub-image and the right pixel sub-image;

[0050] The fused image retains complete brightness and texture information by fusing the left pixel sub-image and the right pixel sub-image. The second network associates the fused image, depth map, disparity map of the current frame with the features of historical frames. It uses the spatial scale and position constraints provided by depth and disparity to correct problems such as pixel value fluctuations caused by illumination changes and feature loss caused by object occlusion. Finally, it outputs an optical flow map that represents the direction and speed of pixel movement. The optical flow map is a vector map that represents the direction and speed of movement of each pixel in the image between consecutive frames.

[0051] By inputting the fused image, the depth map, and the disparity map into a second network, the fused image, the depth map, and the disparity map are analyzed, and an optical flow map of the current shooting scene is output. This overcomes the shortcomings of traditional optical flow methods, which are sensitive to illumination and only applicable to small displacements. It improves the robustness of optical flow estimation in complex scenes, and in particular enhances the ability to capture large displacement motions.

[0052] Step 140: Detect motion information of moving objects in the current shooting scene based on the optical flow map.

[0053] Based on the pixel-level motion vector distribution in the optical flow map, pixel regions with consistent motion trends are identified through cluster analysis, and then the overall motion trajectory, instantaneous velocity, and direction of the moving object are calculated.

[0054] This enables precise analysis of motion information of moving objects. Compared with the limitations of traditional inter-frame difference methods that cannot obtain motion vectors, the direction and velocity information provided by optical flow graphs make motion detection more practical. It can support functions such as adjusting focus parameters in motion capture and calculating compensation in image stabilization, significantly improving the application value of motion detection.

[0055] In one possible embodiment, step 110 may specifically include the following steps:

[0056] Acquire dual-pixel data of the current shooting scene collected by the camera's dual-pixel sensor;

[0057] The dual-pixel data is separated to obtain a left pixel sub-image and a right pixel sub-image.

[0058] Dual-pixel data refers to the raw image data that is simultaneously acquired and output by the left and right sub-pixels of each physical pixel when the dual-pixel sensor of the camera images the current shooting scene. Dual-pixel data contains the light intensity and phase information received by the left and right sub-pixels respectively, and is usually integrated into a frame of raw data in a specific format. Separation processing refers to the process of extracting the image information corresponding to the left sub-pixel array and the image information corresponding to the right sub-pixel array from the dual-pixel data according to the hardware design rules of the dual-pixel sensor, and finally obtaining two independent images.

[0059] Each imaging pixel of a dual-pixel sensor is physically divided into left and right sub-pixels, which have a slight angular difference in space. This can be considered as two miniature perspectives simultaneously capturing scene light. Therefore, the output dual-pixel data is essentially a collection of photosensitive information from the left and right sub-pixels mixed according to preset rules. During separation processing, based on the sub-pixel layout parameters of the sensor at the factory, all pixels belonging to the left sub-pixel in the dual-pixel data are extracted and recombined into a left pixel sub-image through coordinate indexing or masking operations. Similarly, the right sub-pixel is extracted and recombined into a right pixel sub-image, ensuring that the two sub-images correspond one-to-one in spatial position and completely preserving the original phase difference features.

[0060] By acquiring dual-pixel data and performing separation processing, the original disparity information of the left and right sub-pixels can be preserved from the hardware source, avoiding the problem of missing spatial angle information caused by traditional single-pixel sensors only being able to acquire a single image. The separation processing is based on the precise mapping of the sensor's physical structure, ensuring the spatial correspondence of the left and right pixel sub-images, and providing high-quality input data for the subsequent first network to extract phase difference features, calculate depth maps and disparity maps. Even when the target is out of focus, since the phase difference information of the sub-pixels is determined by the hardware structure and is not affected by image sharpness, the separated sub-images can still stably carry the spatial geometric clues of the scene, laying a data foundation for solving the traditional problem of motion detection failure in out-of-focus scenes.

[0061] In one possible embodiment, the first network includes a first sub-network, a second sub-network, an encoder, and a decoder. The first sub-network and the second sub-network have the same network structure and share weights. Step 120 may specifically include the following steps:

[0062] The left pixel sub-image is input into the first sub-network, and feature extraction is performed on the left pixel sub-image to output the high-level features of the left pixel sub-image. The right pixel sub-image is input into the second sub-network, and feature extraction is performed on the right pixel sub-image to output the high-level features of the right pixel sub-image. The high-level features of the left pixel sub-image and the high-level features of the right pixel sub-image are concatenated to obtain concatenated features. The concatenated features are input into the encoder, and feature compression is performed on the concatenated features to obtain compressed features. The compressed features are input into the decoder, and feature reconstruction is performed on the compressed features to output a depth map and a disparity map.

[0063] The first and second sub-networks are parallel network modules within the first network used to extract sub-graph features. They share the same network structure, such as containing the same number of convolutional layers, activation functions, and pooling layers. Weight sharing means that both sub-networks use exactly the same parameters during training and inference, ensuring consistent rules for feature extraction from the left and right pixel sub-graphs. Figure 4 As shown, the first network includes a first sub-network, a second sub-network, an encoder, and a decoder.

[0064] High-level features are abstract features obtained after a sub-image has undergone multiple layers of sub-network processing. They contain semantic information, structural contours, and deep cues related to phase difference, rather than shallow information at the original pixel level.

[0065] The left and right pixel sub-images are respectively input into the first and second sub-networks with the same structure and shared weights. Through convolution operations, features from edges and textures to abstract semantics are extracted layer by layer. Due to the shared weights, the feature mapping methods of the two sub-networks for the left and right sub-images are completely consistent, avoiding feature deviations caused by differences in network parameters. Finally, the high-level features of the left pixel sub-image and the high-level features of the right pixel sub-image are comparable.

[0066] The high-level features of the left pixel sub-image and the high-level features of the right pixel sub-image are concatenated by channel to form a concatenated feature. This feature contains high-level information of both the left and right sub-images as well as clues to their differences, such as feature differences corresponding to phase differences.

[0067] The concatenated feature input encoder is a network module composed of multiple convolutional and downsampling layers. It is used to compress the dimensionality of the input features and extract key information. Through continuous convolution and downsampling operations, it reduces spatial dimensionality, compresses redundant information, and retains the compressed features most relevant to depth and disparity calculation, such as the relative positional deviation between pixels and scene geometry. The compressed features are low-dimensional, high-information-density features obtained by the encoder after processing the concatenated features.

[0068] The compressed features are then processed by a decoder, a network module consisting of deconvolutional and upsampling layers, which reconstructs the compressed features into an output that matches the size of the input sub-image. Spatial resolution is gradually restored through deconvolution and upsampling, and detailed information is supplemented by mechanisms such as residual connections. Finally, a depth map and disparity map with the same size as the original sub-image are reconstructed. The calculation process makes full use of the natural phase difference characteristics of the two-pixel sub-image, and even if the sub-image is blurry, reliable results can still be generated based on stable phase difference features.

[0069] The weight-sharing first and second sub-networks reduce the total number of network parameters and lower training complexity, while ensuring the consistency of features between the left and right sub-images and avoiding disparity calculation bias caused by different feature extraction standards. The concatenation of high-level features effectively integrates the difference information of the two-pixel sub-images, providing richer cues for depth and disparity estimation. The encoder's feature compression removes redundant information and improves the network's computational efficiency, while the decoder's feature reconstruction ensures that the output depth map and disparity map retain sufficient spatial details.

[0070] The overall network structure is optimized for dual-pixel data, breaking through the limitation of traditional depth estimation relying on clear textures. Even if the target is in an out-of-focus and blurry state, it can still generate high-precision depth maps and disparity maps based on phase difference features, providing stable spatial geometric constraints for subsequent motion detection and significantly improving the reliability of motion analysis in complex scenes.

[0071] In one possible embodiment, before the step of inputting the left pixel sub-image and the right pixel sub-image into the first network and extracting features from the left pixel sub-image and the right pixel sub-image to obtain the depth map and disparity map of the current shooting scene, the following steps may also be included:

[0072] Acquire sample data, which includes: a label depth map, a label disparity map, a sample left pixel sub-image, and a sample right pixel sub-image; input the sample data into a sample first network, perform feature extraction on the sample left pixel sub-image and the sample right pixel sub-image to obtain a predicted depth map and a predicted disparity map; train the sample first network based on the label depth map, the predicted depth map, the label disparity map, and the predicted disparity map until a preset training condition is met to obtain the first network.

[0073] The sample data is the dataset used to train the network, containing pairs of input data and corresponding real results; the label depth map is an annotated image of the actual distance from each pixel in the sample scene to the camera, serving as a true reference for depth estimation; the label disparity map is an annotated image of the actual positional deviation of corresponding pixels in the left and right pixel sub-images of the sample, which has a geometric correspondence with the label depth map; the left and right pixel sub-images and the right pixel sub-image of the sample are dual-pixel sub-image data collected from the sample scene and used for training.

[0074] By collecting the true depth and disparity information of the sample scene, a labeled depth map and a labeled disparity map are generated. At the same time, the left and right pixel sub-images of the sample scene are collected simultaneously to construct sample data containing input and labels. The left and right sub-images of the sample are input into the first sample network, which is the first network in the training phase. Its structure is consistent with the final deployed network, but its parameters are not optimized. The predicted depth map and the predicted disparity map are the estimated results output by the sample network after processing the sample sub-images.

[0075] The first sample network extracts and maps internal features to output a predicted depth map and a predicted disparity map. It calculates the difference between the predicted result and the label, and adjusts the network parameters through the backpropagation algorithm to make the predicted result gradually approach the label. The above process is repeated until the preset training conditions are met. The preset training conditions are indicators to judge that the network training is complete. They usually include the loss function value being lower than a threshold, the number of iterations reaching a preset value, or the model performance stabilizing on the validation set. At this time, the sample network has completed training and becomes the first network that can be used in real-world scenarios. Its parameters have learned the mapping rules between the phase difference features of the two-pixel sub-image and the depth and disparity.

[0076] By training with labeled sample data, the first network can learn the complex relationship between phase difference, depth, and disparity hidden in the two-pixel sub-image. Especially for blurred scenes, the network can master the feature extraction rules in the case of high-frequency texture loss by mapping the blurred sub-images in the training samples to the corresponding labels. This overcomes the limitation of traditional algorithms that rely on clear textures, provides an accurate spatial geometric basis for the entire motion detection process, and ultimately improves the motion analysis performance in complex scenes.

[0077] In one possible embodiment, the fused image is obtained by fusing the left and right pixel sub-images of the nth frame from multiple consecutively acquired frames. Step 130 may specifically include the following steps:

[0078] When n equals 1, the fused image, the depth map, and the disparity map of the first frame are processed through the initial hidden state features, and the optical flow map and hidden state features of the first frame are output.

[0079] When n is greater than 1, the fused image, the depth map and the disparity map of the nth frame are processed by combining the hidden state features output from the (n-1)th frame, and the optical flow map and hidden state features of the nth frame are output.

[0080] The hidden state features of the nth frame are used to process the input data of the (n+1)th frame, where n is a positive integer.

[0081] The fused image is generated by pixel-level fusion of the left and right pixel sub-images of the nth consecutively acquired frame, preserving the complete brightness and texture information of the two-pixel sub-images; the initial hidden state features are the initial internal state features used by the second network when processing the first frame of data, usually a preset zero vector or an initialized feature vector, used for temporal modeling when the network starts; the hidden state features are the internal state vectors output by the network after processing each frame of data, containing the key temporal features of the current frame and historical frames, serving as the input context for processing the next frame; n is the frame number, used to identify the order of the consecutively acquired image frames.

[0082] The optical flow graph is a (2, H, W) dimensional tensor, where the first dimension represents the horizontal displacement of the object, the second dimension represents the vertical position, and the positive and negative signs indicate the direction of the object's motion.

[0083] The second network has a built-in temporal recurrent unit that can pass historical information through hidden state features. When processing the first frame (n=1), the second network has no historical frame information and uses the initial hidden state features as the temporal context. The fused image, depth map, and disparity map of the first frame, which are used to provide global scene information, are input into the network. The recurrent unit performs temporal encoding on the input features based on the initial state and outputs the optical flow map and hidden state features of the first frame.

[0084] When processing the nth frame (n>1), the network structure of the second network is as follows: Figure 5 As shown, by calling the hidden state features output from frame (n-1) as the temporal context of the current frame, and inputting them together with the fused image, depth map, and disparity map of frame n into the recurrent unit, the motion coherence between frames is captured through interactive modeling of historical features and current features, and finally outputs the optical flow map of frame n and the updated hidden state features. The updated hidden state features will continue to be passed to frame n+1, forming a temporal closed loop.

[0085] like Figure 6 As shown, the moving object only has a positive displacement in the horizontal direction, and the displacement is s pixels. In the two-dimensional motion vector of the optical flow graph, the first dimension vector value of the moving object is s, and the horizontal vector value of the other positions is 0. Since the object does not have a vertical displacement, the second dimension vector value of all positions in the optical flow graph is 0.

[0086] By passing hidden state features between frames, the second network can effectively model the temporal correlation of consecutive frames, overcoming the problem of motion continuity loss caused by traditional optical flow methods relying only on a single frame or two adjacent frames. In particular, it improves the optical flow estimation accuracy for large displacement motion and velocity change scenarios. The setting of the initial hidden state features ensures the stability of the first frame processing and avoids estimation deviation when there is no historical information.

[0087] The spatial geometric constraints provided by depth maps and disparity maps, combined with temporal hidden states, further enhance the network's robustness to changes in lighting and object occlusion. For example, when a region is temporarily invisible due to occlusion, the second network can predict its current position through motion trajectory information in historical hidden states, reducing the generation of abnormal optical flow vectors. The final output optical flow map can more accurately reflect the continuous motion state of the target, providing a more reliable vector basis for subsequent motion detection and improving the real-time performance and accuracy in functions such as motion capture and video stabilization.

[0088] In one possible embodiment, the hidden state features are generated by a second network from the input sequence of frame 1 to frame (n-1); the hidden state features include a fused feature of the following characteristics:

[0089] Depth map features of at least some frames from frame 1 to frame n-1, disparity map features of at least some frames from frame 1 to frame n-1, and motion trajectory features of at least some frames from frame 1 to frame n-1.

[0090] Hidden state features are internal state vectors maintained by the second network through temporal recurrent units when processing continuous frame data. They contain key information from historical frames and serve as the context for processing the current frame. The second network is a deep learning model that integrates recurrent structures, enabling temporal modeling of multi-frame input sequences. Depth map features are spatial distance-related features extracted from the depth map. Disparity map features are pixel position deviation-related features obtained by feature extraction from the disparity map. Motion trajectory features are target pixel motion path features calculated from historical optical flow maps.

[0091] When processing frames 1 to n-1, the second network uses a recurrent unit state update mechanism to gradually fuse the depth map features, disparity map features, and motion trajectory features of each frame into the hidden state. Specifically, the depth map features and disparity map features are extracted through convolutional or fully connected layers, and then channel-wise concatenated or weighted fused with the motion trajectory features calculated from the optical flow map to form a composite feature vector containing both spatial geometric information and temporal motion information.

[0092] The recurrent unit selectively retains long-term effective features and updates short-term changing features, allowing the hidden state to gradually accumulate spatiotemporal correlation information across multiple frames as the input sequence progresses. When processing the nth frame, the second network inputs the fused image, depth map, disparity map of the current frame, and hidden state features containing information from multiple historical frames. Through temporal reasoning by the recurrent unit, combined with historical spatial constraints and motion patterns, it optimizes the estimation of the optical flow map for the current frame.

[0093] By fusing depth map features, disparity map features, and motion trajectory features from multiple frames, the hidden state features not only contain local spatial information of the current frame but also encode the target's motion pattern in the temporal dimension. This solves the problem of large displacement motion estimation bias caused by traditional optical flow methods relying solely on two adjacent frames. The long-term accumulation of depth and disparity features enables the second network to capture the static structure of the scene. When a dynamic target is partially occluded, it can distinguish between real motion and occlusion artifacts through stable depth information in historical hidden states, reducing the generation of abnormal optical flow vectors.

[0094] The incorporation of motion trajectory features enhances the ability to model the continuity of target motion. For example, in sports photography, even if a frame is partially blurred due to the athlete's rapid movement, the second network can still accurately predict the motion vector of the current frame using the motion trajectory features stored in the historical hidden states, improving the robustness and accuracy of optical flow estimation in complex motion scenes. Ultimately, this provides more comprehensive spatiotemporal context information for motion detection, improving the accuracy of motion detection in dynamic and complex environments.

[0095] In one possible embodiment, after step 140, the following steps may also be included:

[0096] If the motion information of the moving object includes vertical displacement, the motor of the camera is driven to perform a focusing operation;

[0097] If the motion information of the moving object does not include vertical displacement, a power reduction operation is performed.

[0098] Motion information of a moving object refers to the motion parameters of the moving object between consecutive frames obtained through optical flow graph analysis, including displacement direction, speed, and trajectory. Vertical displacement specifically refers to the change in distance of the moving object along the optical axis of the camera, which directly affects image clarity. The camera's motor can be a voice coil motor, which is the core execution component that drives the lens module to adjust its position to achieve focusing. Focusing operation refers to the process of the motor driving the lens to change the focal length so that the moving object is clearly imaged on the sensor. Power reduction operation refers to strategies to reduce the power consumption of camera-related modules when high-frequency hardware actions are not required, such as reducing the number of motor drives and reducing the processor's calculation frequency for motion detection.

[0099] Based on the motion information of the moving object, the system first determines whether there is vertical displacement. Since vertical displacement reflects the change in distance between the target and the camera, if the focus is not adjusted, the moving object will be out of focus, resulting in blurred images. Therefore, the camera motor needs to be triggered to perform a focusing operation. The motor drives the lens to fine-tune its position and compensate for the change in object distance to restore focus. If there is no vertical displacement in the motion information, it means that the distance between the target and the camera is stable and there is no need for frequent focusing. At this time, the system can perform power-saving operations, such as stopping the motor drive, reducing the frame rate of the motion detection algorithm, or reducing the power supply voltage of related chips, thereby reducing energy consumption while ensuring the current image quality.

[0100] By dynamically adjusting camera operation based on whether vertical displacement occurs, this system ensures real-time focusing capability even when the target distance changes, preventing blurry images caused by defocusing during motion capture or video recording, thus improving image quality. Furthermore, in scenarios where focusing is not required, it reduces unnecessary energy consumption through power-saving operations, alleviating the power consumption problem of the camera module in electronic devices and extending device battery life. This refined control based on motion information achieves a balance between imaging performance and power consumption. Especially when dynamic and static scenes alternate, it can adaptively adjust hardware states, significantly improving the camera's intelligence level and user experience.

[0101] Specifically, the step of performing power reduction operation when the motion information of the moving object does not include vertical displacement may include the following steps:

[0102] If the motion information of the moving object does not include vertical displacement, the calculation frame rate of the autofocus statistics is adjusted from a first frequency to a second frequency, where the first frequency is greater than the second frequency.

[0103] Autofocus statistics are a set of key parameters used by a camera system to evaluate focus status. They typically include phase difference amplitude, image contrast gradient, edge sharpness, etc. The calculation results of autofocus statistics directly affect the driving decisions of the focus module. The calculation frame rate refers to the number of times the autofocus statistics are calculated per unit time. The first frequency is the high-frequency calculation rate used by the system in high dynamic scenes, such as 50Hz, and the second frequency is the low-frequency calculation rate in low dynamic scenes, such as 20Hz. The two satisfy the setting relationship that the first frequency is greater than the second frequency.

[0104] When the motion information of the moving object does not include vertical displacement, the distance between the camera and the target remains stable, and the focusing module does not need to update the focusing parameters frequently. By detecting the motion vector of the moving object in the optical flow graph, it is determined whether there is a displacement component along the optical axis of the camera. If not, a power reduction operation is triggered, and the calculation frame rate of the autofocus statistics is adjusted from the first frequency to the second frequency.

[0105] The adjustment from the first frequency to the second frequency is achieved by modifying the timer interrupt parameters of the driver layer or the processor task scheduling strategy. For example, the statistical calculation task, which originally executed every 20ms, is extended to execute every 50ms, reducing the amount of computation per unit time. Since zero vertical displacement means that the current focus state is stable, reducing the calculation frame rate will not affect the focus accuracy and can significantly reduce the computing power consumption of the image signal processor or neural network processor.

[0106] By dynamically adjusting the calculation frequency of autofocus statistics, in scenarios where the target distance remains constant, the computing resources originally used for high-frequency focusing calculations are released, allowing the processor to enter a low-load state, thereby reducing the overall power consumption of the camera module. For example, adjusting the frequency from the first to the second reduces the number of focusing calculations, and combined with the sleep strategy of the motor drive circuit, this lowers the power consumption of the camera module. Through this power consumption control strategy, image quality in stable focusing scenarios is ensured while avoiding the energy waste caused by traditional fixed frame rate calculations. This is particularly suitable for scenarios such as video recording and long-term framing, effectively extending the battery life of electronic devices without affecting the user experience.

[0107] In the embodiments of this application, by acquiring the left and right pixel sub-images of the current shooting scene collected by the dual-pixel sensor of the camera, the spatial angle difference information contained in the dual-pixel sub-images provides data support for solving motion detection in out-of-focus scenes. The left and right pixel sub-images are input into a first network to extract features from them. Utilizing the phase difference characteristics of the dual-pixel sub-images, reliable depth maps and disparity maps can still be generated even if the target is blurred, providing key spatial geometric constraints for subsequent motion analysis. The fused image, depth map, and disparity map are input into a second network. The fused image is obtained by fusing the left and right pixel sub-images, preserving the complete brightness and texture information of the dual pixels. The second network is a deep learning model containing a temporal recurrent structure, which can jointly analyze the fused image, depth map, and disparity map of the current frame using historical frame information, and outputs an optical flow map to characterize the motion direction and velocity vector of pixels between consecutive frames, improving the robustness of optical flow estimation in complex scenes. Based on the optical flow map, the motion information of moving objects in the current shooting scene can be detected, which can improve the accuracy of motion detection.

[0108] The motion detection method provided in this application can be executed by a motion detection device. This application uses an example of a motion detection device executing the motion detection method to illustrate the motion detection device provided in this application.

[0109] Figure 7 This is a block diagram of a motion detection device provided in an embodiment of this application. The device 700 includes:

[0110] The acquisition module 710 is used to acquire the left and right pixel sub-images of the current shooting scene captured by the dual-pixel sensor of the camera;

[0111] The first input module 720 is used to input the left pixel sub-image and the right pixel sub-image into the first network, extract features from the left pixel sub-image and the right pixel sub-image, and obtain the depth map and disparity map corresponding to the current shooting scene;

[0112] The second input module 730 is used to input the fused image, the depth map, and the disparity map into the second network, analyze the fused image, the depth map, and the disparity map, and output the optical flow map of the current shooting scene; the fused image is obtained by fusing the left pixel sub-image and the right pixel sub-image;

[0113] The detection module 740 is used to determine the motion information of moving objects in the current shooting scene based on the optical flow map.

[0114] In one possible embodiment, the first network includes a first sub-network, a second sub-network, an encoder, and a decoder. The first sub-network and the second sub-network have the same network structure and share weights. The first input module 720 is specifically used for:

[0115] The left pixel sub-image is input into the first sub-network, features are extracted from the left pixel sub-image, and the high-level features of the left pixel sub-image are output.

[0116] The right pixel sub-image is input into the second sub-network, features are extracted from the right pixel sub-image, and the high-level features of the right pixel sub-image are output.

[0117] The high-level features of the left pixel sub-image and the high-level features of the right pixel sub-image are concatenated to obtain the concatenated features;

[0118] The splicing features are input into the encoder, and the splicing features are compressed to obtain compressed features;

[0119] The compressed features are input into the decoder to reconstruct the features and output a depth map and a disparity map.

[0120] In one possible embodiment, the second input module 730 is specifically used for:

[0121] When n equals 1, the fused image, the depth map, and the disparity map of the first frame are processed through the initial hidden state features, and the optical flow map and hidden state features of the first frame are output.

[0122] When n is greater than 1, the fused image, the depth map and the disparity map of the nth frame are processed by combining the hidden state features output from the (n-1)th frame, and the optical flow map and hidden state features of the nth frame are output.

[0123] The hidden state features of the nth frame are used to process the input data of the (n+1)th frame, where n is a positive integer.

[0124] In one possible embodiment, the hidden state features are generated by the second network from the input sequence of frame 1 to frame (n-1); the hidden state features include a fusion feature of the following features:

[0125] Depth map features of at least some frames from frame 1 to frame n-1, disparity map features of at least some frames from frame 1 to frame n-1, and motion trajectory features of at least some frames from frame 1 to frame n-1.

[0126] In one possible embodiment, the acquisition module 710 is specifically used for:

[0127] Acquire dual-pixel data of the current shooting scene collected by the dual-pixel sensor of the camera;

[0128] The dual-pixel data is separated to obtain a left pixel sub-image and a right pixel sub-image.

[0129] In one possible embodiment, the device 700 may further include:

[0130] A drive module is used to drive the motor of the camera to perform a focusing operation when the motion information of the moving object includes vertical displacement.

[0131] An execution module is used to perform a power reduction operation when the motion information of the moving object does not include vertical displacement.

[0132] In one possible embodiment, the execution module is specifically configured to adjust the calculation frame rate of the autofocus statistics from a first frequency to a second frequency, wherein the first frequency is greater than the second frequency, when the motion information of the moving object does not include vertical displacement.

[0133] In the embodiments of this application, by acquiring the left and right pixel sub-images of the current shooting scene collected by the dual-pixel sensor of the camera, the spatial angle difference information contained in the dual-pixel sub-images provides data support for solving motion detection in out-of-focus scenes. The left and right pixel sub-images are input into a first network to extract features from them. Utilizing the phase difference characteristics of the dual-pixel sub-images, reliable depth maps and disparity maps can still be generated even if the target is blurred, providing key spatial geometric constraints for subsequent motion analysis. The fused image, depth map, and disparity map are input into a second network. The fused image is obtained by fusing the left and right pixel sub-images, preserving the complete brightness and texture information of the dual pixels. The second network is a deep learning model containing a temporal recurrent structure, which can jointly analyze the fused image, depth map, and disparity map of the current frame using historical frame information, and outputs an optical flow map to characterize the motion direction and velocity vector of pixels between consecutive frames, improving the robustness of optical flow estimation in complex scenes. Based on the optical flow map, the motion information of moving objects in the current shooting scene can be detected, which can improve the accuracy of motion detection.

[0134] The motion detection device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific devices.

[0135] The motion detection device in this application embodiment can be a device with a motion system. The motion system can be an Android motion system, an iOS motion system, or other possible motion systems; this application embodiment does not specifically limit it.

[0136] The motion detection device provided in this application embodiment can realize the various processes implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0137] Optionally, such as Figure 8 As shown, this application embodiment also provides an electronic device 810, including a processor 811, a memory 812, and a program or instructions stored in the memory 812 and executable on the processor 811. When the program or instructions are executed by the processor 811, they implement the various steps of any of the above motion detection method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0138] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0139] Figure 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0140] The electronic device 900 includes, but is not limited to, components such as: radio frequency unit 901, network module 902, audio output unit 903, input unit 904, sensor 905, display unit 906, user input unit 907, interface unit 908, memory 909, and processor 910.

[0141] Those skilled in the art will understand that the electronic device 900 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 910 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0142] The processor 910 is used to acquire the left and right pixel sub-images of the current shooting scene captured by the dual-pixel sensor of the camera.

[0143] The processor 910 is also configured to input the left pixel sub-image and the right pixel sub-image into the first network, extract features from the left pixel sub-image and the right pixel sub-image, and obtain a depth map and a disparity map corresponding to the current shooting scene;

[0144] The processor 910 is further configured to input the fused image, the depth map, and the disparity map into a second network, analyze the fused image, the depth map, and the disparity map, and output an optical flow map of the current shooting scene; the fused image is obtained by fusing the left pixel sub-image and the right pixel sub-image;

[0145] The processor 910 is also configured to determine motion information of moving objects in the current shooting scene based on the optical flow map.

[0146] Optionally, the first network includes a first sub-network, a second sub-network, an encoder, and a decoder. The first sub-network and the second sub-network have the same network structure and share weights. The processor 910 is further configured to input the left pixel sub-image into the first sub-network, extract features from the left pixel sub-image, and output the high-level features of the left pixel sub-image.

[0147] The processor 910 is further configured to input the right pixel sub-image into the second sub-network, extract features from the right pixel sub-image, and output the high-level features of the right pixel sub-image;

[0148] The processor 910 is also used to concatenate the high-level features of the left pixel sub-image and the high-level features of the right pixel sub-image to obtain concatenated features.

[0149] The processor 910 is also configured to input the splicing features to the encoder, perform feature compression on the splicing features, and obtain compressed features;

[0150] The processor 910 is also configured to input the compressed features to the decoder, perform feature reconstruction on the compressed features, and output a depth map and a disparity map.

[0151] Optionally, the processor 910 is further configured to process the fused image, the depth map, and the disparity map of the first frame through the initial hidden state features when n equals 1, and output the optical flow map and hidden state features of the first frame;

[0152] The processor 910 is also configured to, when n is greater than 1, process the fused image, the depth map and the disparity map of the nth frame in combination with the hidden state features output in the (n-1)th frame, and output the optical flow map and hidden state features of the nth frame;

[0153] The hidden state features of the nth frame are used to process the input data of the (n+1)th frame, where n is a positive integer.

[0154] Optionally, the hidden state features are generated by the second network from the input sequence of frame 1 to frame (n-1); the hidden state features include a fusion of the following features:

[0155] Depth map features of at least some frames from frame 1 to frame n-1, disparity map features of at least some frames from frame 1 to frame n-1, and motion trajectory features of at least some frames from frame 1 to frame n-1.

[0156] Optionally, the processor 910 is also configured to acquire dual-pixel data of the current shooting scene collected by the dual-pixel sensor of the camera;

[0157] The processor 910 is also used to perform separation processing on the dual-pixel data to obtain a left pixel sub-image and a right pixel sub-image.

[0158] Optionally, the processor 910 is also configured to drive the motor of the camera to perform a focusing operation when the motion information of the moving object includes vertical displacement.

[0159] The processor 910 is also configured to perform a power reduction operation when the motion information of the moving object does not include vertical displacement.

[0160] Optionally, the processor 910 is further configured to adjust the calculation frame rate of the autofocus statistics from a first frequency to a second frequency, wherein the first frequency is greater than the second frequency, when the motion information of the moving object does not include vertical displacement.

[0161] In the embodiments of this application, by acquiring the left and right pixel sub-images of the current shooting scene collected by the dual-pixel sensor of the camera, the spatial angle difference information contained in the left and right pixel sub-images provides data support for solving motion detection in out-of-focus scenes. The left and right pixel sub-images are input into the first network to extract features from them. By utilizing the phase difference characteristics of the dual-pixel sub-images, even if the target is blurred, reliable depth maps and disparity maps can still be generated, providing key spatial geometric constraints for subsequent motion analysis.

[0162] The fused image, depth map, and disparity map are input into the second network. The fused image is obtained by fusing the left and right pixel sub-images, preserving the complete brightness and texture information of the two pixels. The second network is a deep learning model containing a temporal recurrent structure, which can jointly analyze the fused image, depth map, and disparity map of the current frame using historical frame information. It outputs an optical flow map to characterize the motion direction and velocity vector of pixels between consecutive frames, improving the robustness of optical flow estimation in complex scenes. Based on the optical flow map, the motion information of moving objects in the current shooting scene can be detected, which can improve the accuracy of motion detection.

[0163] It should be understood that, in this embodiment, the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042. The GPU 9041 processes image data of still images or video images obtained by an image capture device (such as a camera) in video image capture mode or image capture mode. The display unit 906 may include a display panel 9061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here. The memory 909 can be used to store software programs and various data, including but not limited to applications and motion systems. Processor 910 can integrate an application processor and a modem processor. The application processor mainly handles the action system, user page, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 910.

[0164] The memory 909 can be used to store software programs and various data. The memory 909 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 909 may include volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 909 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0165] Processor 910 may include one or more processing units; optionally, processor 910 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 910.

[0166] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described motion detection method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0167] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0168] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described motion detection method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0169] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0170] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the motion detection method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0171] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0173] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A motion detection method, characterized in that, The method includes: Obtain the left and right pixel sub-images of the current shooting scene captured by the camera's dual-pixel sensor; The left pixel sub-image and the right pixel sub-image are input into the first network, and feature extraction is performed on the left pixel sub-image and the right pixel sub-image to obtain the depth map and disparity map corresponding to the current shooting scene; The fused image, the depth map, and the disparity map are input into a second network. The fused image, the depth map, and the disparity map are analyzed, and an optical flow map of the current shooting scene is output. The fused image is obtained by fusing the left pixel sub-image and the right pixel sub-image. Based on the optical flow diagram, the motion information of moving objects in the current shooting scene is determined.

2. The method according to claim 1, characterized in that, The first network includes a first sub-network, a second sub-network, an encoder, and a decoder. The first sub-network and the second sub-network have the same network structure and share weights. The step of inputting the left pixel sub-image and the right pixel sub-image into the first network, and extracting features from the left pixel sub-image and the right pixel sub-image to obtain the depth map and disparity map corresponding to the current shooting scene includes: The left pixel sub-image is input into the first sub-network, features are extracted from the left pixel sub-image, and the high-level features of the left pixel sub-image are output. The right pixel sub-image is input into the second sub-network, features are extracted from the right pixel sub-image, and the high-level features of the right pixel sub-image are output. The high-level features of the left pixel sub-image and the high-level features of the right pixel sub-image are concatenated to obtain the concatenated features; The splicing features are input into the encoder, and the splicing features are compressed to obtain compressed features; The compressed features are input into the decoder to reconstruct the features and output a depth map and a disparity map.

3. The method according to claim 1, characterized in that, The fused image is obtained by fusing the left and right pixel sub-images of the nth frame from multiple consecutively acquired frames. The analysis of the fused image, the depth map, and the disparity map to output the optical flow map of the current shooting scene includes: When n equals 1, the fused image, the depth map, and the disparity map of the first frame are processed through the initial hidden state features, and the optical flow map and hidden state features of the first frame are output. When n is greater than 1, the fused image, the depth map, and the disparity map of the nth frame are processed by combining the hidden state features output from the (n-1)th frame, and the optical flow map and hidden state features of the nth frame are output. The hidden state features of the nth frame are used to process the input data of the (n+1)th frame, where n is a positive integer.

4. The method according to claim 3, characterized in that, The hidden state features are generated by the second network from the input sequence of frame 1 to frame (n-1); the hidden state features include a fusion feature of the following features: Depth map features of at least some frames from frame 1 to frame n-1, disparity map features of at least some frames from frame 1 to frame n-1, and motion trajectory features of at least some frames from frame 1 to frame n-1.

5. The method according to claim 1, characterized in that, The acquisition of the left and right pixel sub-images of the current shooting scene captured by the camera's dual-pixel sensor includes: Acquire dual-pixel data of the current shooting scene collected by the dual-pixel sensor of the camera; The dual-pixel data is separated to obtain a left pixel sub-image and a right pixel sub-image.

6. The method according to claim 1, characterized in that, After determining the motion information of the moving object in the current shooting scene based on the optical flow map, the method further includes: If the motion information of the moving object includes vertical displacement, the motor of the camera is driven to perform a focusing operation; If the motion information of the moving object does not include vertical displacement, a power reduction operation is performed.

7. The method according to claim 6, characterized in that, When the motion information of the moving object does not include vertical displacement, the power reduction operation includes: If the motion information of the moving object does not include vertical displacement, the calculation frame rate of the autofocus statistics is adjusted from a first frequency to a second frequency, where the first frequency is greater than the second frequency.

8. A motion detection device, characterized in that, The device includes: The acquisition module is used to acquire the left and right pixel sub-images of the current shooting scene captured by the dual-pixel sensor of the camera; The first input module is used to input the left pixel sub-image and the right pixel sub-image into the first network, and to extract features from the left pixel sub-image and the right pixel sub-image to obtain the depth map and disparity map corresponding to the current shooting scene; The second input module is used to input the fused image, the depth map, and the disparity map into the second network, analyze the fused image, the depth map, and the disparity map, and output the optical flow map of the current shooting scene; the fused image is obtained by fusing the left pixel sub-image and the right pixel sub-image; The detection module is used to determine the motion information of moving objects in the current shooting scene based on the optical flow map.

9. The apparatus according to claim 8, characterized in that, The first network includes a first sub-network, a second sub-network, an encoder, and a decoder. The first sub-network and the second sub-network have the same network structure and share weights. The first input module is specifically used for: The left pixel sub-image is input into the first sub-network, features are extracted from the left pixel sub-image, and the high-level features of the left pixel sub-image are output. The right pixel sub-image is input into the second sub-network, features are extracted from the right pixel sub-image, and the high-level features of the right pixel sub-image are output. The high-level features of the left pixel sub-image and the high-level features of the right pixel sub-image are concatenated to obtain the concatenated features; The splicing features are input into the encoder, and the splicing features are compressed to obtain compressed features; The compressed features are input into the decoder to reconstruct the features and output a depth map and a disparity map.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1 to 7.

Images