Lens mode detection method and related apparatus, electronic device and storage medium

Abstract

This application discloses a lens pattern detection method, related apparatus, electronic device, and storage medium. The lens pattern detection method includes: performing scene detection based on at least one image set to obtain scene detection results; wherein the image set includes a first image and a second image captured sequentially by a camera device, and the scene detection results characterize the probability that the second image in each image set undergoes a scene change relative to the first image; filtering the image set based on the scene detection results to obtain several target images, and constructing a test video segment based on the several target images; and determining the lens pattern of the camera device based on the test video segment. This scheme can detect the lens pattern of the camera device as accurately as possible.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a lens pattern detection method and related apparatus, electronic device and storage medium. Background Technology

[0002] Currently, people have higher requirements for the performance of camera devices, so camera devices also have complex functions such as a shooting range of up to 360 degrees, autofocus, auto zoom, and automatic video image capture.

[0003] However, video streams captured by cameras after lens shifting, tilting, or zooming at a fixed point present significant challenges for backend video analysis equipment in tasks such as video clipping and reuse, and video event analysis, especially when the current lens mode of the camera is unknown. Therefore, accurately detecting the current lens mode of the camera has become a pressing issue. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a lens mode detection method and related devices, electronic equipment, and storage medium that can detect the lens mode of a camera device as accurately as possible.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a lens mode detection method, comprising: performing scene detection based on at least one image set to obtain scene detection results; wherein the image set includes a first image and a second image captured sequentially by a camera device, and the scene detection results characterize the probability that the second image in each image set undergoes a scene change relative to the first image; filtering the image set based on the scene detection results to obtain several target images, and constructing a video segment to be tested based on the several target images; and determining the lens mode of the camera device based on the video segment to be tested.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a lens mode detection device, comprising: a scene detection module, a video construction module, and a mode determination module. The scene detection module performs scene detection based on at least one image set to obtain scene detection results; wherein the image set includes a first image and a second image captured sequentially by a camera device, and the scene detection results characterize the probability that the second image in each image set undergoes a scene change relative to the first image; the video construction module filters the at least one image set based on the scene detection results to obtain several target images, and constructs a video segment to be tested based on the several target images; the mode determination module determines the lens mode of the camera device based on the video segment to be tested.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device including a processor and a memory, wherein the memory and the processor are coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the lens pattern detection method in the first aspect.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the lens pattern detection method described in the first aspect.

[0009] In the above scheme, scene detection is first performed on the image set to obtain scene detection results. Then, target images with a high probability of scene change are selected based on the scene detection results. These target images are used to construct test video segments, which are then classified to determine the camera mode. Because scene detection is performed at the image set level to obtain the probability of scene change in the image set and target images are selected accordingly, the test video segments constructed from the target images also have significant scene change characteristics. Therefore, analysis based on significant scene change characteristics can determine the lens mode of the camera device when shooting the test video segments as accurately as possible. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the lens pattern detection method of this application;

[0011] Figure 2 This is a schematic diagram of extracting images from a video stream at fixed frame intervals;

[0012] Figure 3 This is a flowchart illustrating an embodiment of the steps for obtaining the first and second sample image sets.

[0013] Figure 4 This is a schematic diagram of the second sample image set;

[0014] Figure 5 This is a flowchart illustrating one embodiment of the scene detection model training step;

[0015] Figure 6 yes Figure 1 A flowchart illustrating an embodiment of step S11;

[0016] Figure 7 This is a schematic diagram of an image buffer unit with a single-buffered first-in-first-out structure.

[0017] Figure 8 This is a schematic diagram of an image cache unit with a double-buffered ping-pong structure.

[0018] Figure 9This is a schematic diagram of the framework of an embodiment of the pattern detection model;

[0019] Figure 10 This is a flowchart illustrating one embodiment of the training steps for a pattern detection model.

[0020] Figure 11 This is a schematic diagram of a sample video clip;

[0021] Figure 12 yes Figure 1 A flowchart illustrating an embodiment of step S13;

[0022] Figure 13 This is a schematic diagram of the frame of an embodiment of the lens pattern detection device of this application;

[0023] Figure 14 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0024] Figure 15 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0025] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0026] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0027] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0028] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the lens pattern detection method of this application. Specifically, the lens pattern detection method in this embodiment may include the following steps:

[0029] Step S11: Perform scene detection based on at least one image set to obtain scene detection results.

[0030] In this embodiment, the image set includes a first image and a second image captured sequentially by a camera device. The scene detection result characterizes the probability that the second image in each image set undergoes a scene change relative to the first image.

[0031] In one implementation scenario, the application of the camera device has high requirements for real-time information (e.g., real-time video monitoring). In this case, the video stream information of the camera device can be acquired in real time, and adjacent images in the video stream information can be combined to form an image set.

[0032] In another implementation scenario, where the application of the camera device has lower requirements for real-time information (e.g., remote video transmission), the video stream information of the camera device can be acquired at intervals, and then images can be extracted and constructed into an image set according to a fixed frame interval. Alternatively, the implementation can also acquire the video stream information of the camera device and then extract images and construct an image set according to a fixed frame interval. Figure 2 The above, Figure 2 This is a schematic diagram of extracting images from a video stream at fixed frame intervals.

[0033] In one implementation scenario, due to the differences between various camera devices, the video stream information and the image set also differ. In order to minimize redundant calculations, the image captured in a single shot can be fixed by scaling operations so that the image size is a fixed value.

[0034] In one implementation scenario, to improve the efficiency and accuracy of scene detection, deep learning technology can be applied to construct a scene detection model. Scene detection is performed by this model, which can be trained based on a first set of sample images and a second set of sample images. The first set of sample images includes first and second sample images, where the second sample image shows no scene change compared to the first sample image. The second set of sample images includes a third and a fourth sample image, where the fourth sample image shows a scene change compared to the third sample image. Therefore, by constructing a scene detection model using deep learning technology, the scene detection method can be continuously optimized with a large number of sample images, resulting in more accurate scene detection results and facilitating subsequent determination of shot modes. Specifically, the scene detection model can include a feature extraction network and a feature contrast network, both of which analyze and process the images to achieve the scene detection function.

[0035] Further, please refer to Figure 3 , Figure 3 This is a schematic flowchart illustrating an embodiment of the steps for obtaining the first and second sample image sets. Specifically, it may include the following steps:

[0036] Step S301: Select the set of images from several sample images that have not undergone scene changes to form the first sample image set.

[0037] In one implementation scenario, historical video data from a camera device can be acquired and images extracted. From the extracted images, two images—a second sample image and a first sample image—that show no scene change compared to the first sample image are selected to form the first sample image set.

[0038] In another implementation scenario, images captured by devices such as cameras and mobile phones can be used as sample images, and two images that have not changed in scene compared to the first sample image can be selected from the second sample image to form the first sample image set.

[0039] Step S302: Perform image filtering based on several sample images to obtain a second sample image set, and / or perform image enhancement based on the first sample image set to obtain a second sample image set.

[0040] It should be noted that the second sample image set is the set of images where the scene has changed, meaning the fourth sample image differs from the third sample image in terms of scene. The second sample image can be obtained by filtering the sample images, or it can be obtained by enhancing the first sample image. Therefore, by constructing two sets of sample images, representing stable and changed scenes respectively, the scene detection model can perform deep learning of scene features under both conditions, effectively improving its detection performance.

[0041] In a specific implementation scenario, if a sufficient number of sample images are acquired, scene change filtering can be performed on the sample images to select those with scene changes, forming a second sample image set. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of the second sample image set. (For example...) Figure 4 As shown, image set 4a represents the image set rotated vertically up and down, image set 4b represents the image set stretched by the lens, and image set 4c represents the image set rotated horizontally left and right.

[0042] In another specific implementation scenario, if there are few or insignificant images with changes in the sample image scene, image enhancement processing can be performed on the first sample image set. The image enhancement processing methods include randomly combining images by rotation, magnification, center radial changes, left-right and top-bottom offsets, etc., to obtain the second sample image set.

[0043] Of course, the two methods for obtaining the second sample image set mentioned above can also be used in combination. For specific implementation methods, please refer to the aforementioned methods, which will not be repeated here.

[0044] Furthermore, after obtaining the first and second sample image sets, a scene detection model can be trained based on the first and second sample image sets. Please refer to [link / reference]. Figure 5 , Figure 5 This is a flowchart illustrating one embodiment of the scene detection model training step. Specifically, training the scene detection model in the above embodiment may include the following steps:

[0045] Step S501: Extract features from the first sample image set and the second sample image set respectively to obtain the first sample background features of the first sample image, the second sample background features of the second sample image, the third sample background features of the third sample image, and the fourth sample background features of the fourth sample image.

[0046] In one implementation scenario, to improve the efficiency of model training, the first and second sample image sets can be normalized before feature extraction to meet the input requirements of the scene detection model. As mentioned earlier, the scene detection model includes a feature extraction network used to extract features from the first and second sample image sets, obtaining background features of the first, second, third, and fourth samples. The extracted feature vectors serve as an abstract semantic representation of the sample image sets.

[0047] In a specific implementation scenario, the feature extraction network can be implemented using 2D convolutional layers, nonlinear activation layers, or residual structure blocks composed of these two basic structures. The scheme does not specifically limit the chosen network type. The feature vector of the first sample image set is denoted as (v1...). 稳 v2 稳 The feature vector of the second sample image set is denoted as (v1). 变 v2 变 Among them, v1 稳 v2 represents the background features of the first sample. 稳 v1 represents the background features of the second sample. 变 v2 represents the background features of the third sample. 变 This represents the background features of the fourth sample.

[0048] Considering that the original feature vectors extracted are three-dimensional vectors, they can be reshaped to facilitate subsequent analysis and calculation. For example, if the original feature dimension is 3*3*128, the reshaped vector can be 1*1152. Of course, the specific feature dimension mentioned above is only one possible implementation in practical application scenarios and does not limit the specific value of the feature dimension.

[0049] Step S502: Make a prediction based on the background features of the first sample and the background features of the second sample to obtain the detection result of the first sample, and / or make a prediction based on the background features of the third sample and the background features of the fourth sample to obtain the detection result of the second sample.

[0050] In this embodiment, the first sample detection result represents the probability that the first sample image has undergone a scene change relative to the second sample image, and the second sample detection result represents the probability that the third sample image has undergone a scene change relative to the fourth sample image. In practical applications, to maximize the accuracy of the scene detection model, the first sample detection result can be obtained simultaneously based on the first sample background features and the second sample background features, and the second sample detection result can be obtained based on the third sample background features and the fourth sample background features; alternatively, the first sample detection result can be obtained based on both the first and second sample background features; or, the second sample detection result can be obtained based on both the third and fourth sample background features. For example, to maximize the accuracy of the scene detection model, the scene detection model can be constrained simultaneously by the first and second sample detection results, which helps improve prediction performance.

[0051] In one implementation scenario, as described above, the scene detection model includes a feature comparison network, which is used to compare the background features of the first sample and the background features of the second sample to obtain the detection result of the first sample, and / or to predict the background features of the third sample and the background features of the fourth sample to obtain the detection result of the second sample.

[0052] In a specific implementation scenario, the feature comparison network consists of fully connected layers and nonlinear pooling. This application does not impose specific limitations on the number and size of the fully connected units and nonlinear pooling units in the middle of the network. The final output dimension of the overall feature comparison network must be guaranteed to be 1*1.

[0053] In another specific implementation scenario, for the convenience of feature comparison, the Manhattan distance can be calculated on the feature vectors of the first sample image set and the feature vectors of the second sample image set before being fed into the feature comparison network for feature comparison. The Manhattan distance calculation formula includes the following formulas (1) and (2).

[0054] d 稳 =|v1 稳 -v2 稳 |……(1)

[0055] d 变 =|v1 变 -v2 变 |……(2)

[0056] Among them, v1 稳 v2 represents the background features of the first sample. 稳 v1 represents the background features of the second sample. 变 v2 represents the background features of the third sample. 变 d represents the background features of the fourth sample. 稳d represents the Manhattan distance of the feature vectors of the first sample image set. 变 This represents the Manhattan distance between the feature vectors of the second sample image set. Furthermore, in practical applications, other distance metrics such as Euclidean distance and Chebyshev distance can also be used to measure the feature distance; this is not a limitation here.

[0057] In another specific implementation scenario, the detection results of the first sample and the detection results of the second sample can be probability-mapped. For example, the probability z1 of the second sample image in the first sample image set changing scene relative to the first sample image and the probability z2 of the fourth sample image in the second sample image set changing scene relative to the third sample image can be output by the sigmoid unit.

[0058] Step S503: Based on the feature distance between the background features of the first sample, the background features of the second sample, the background features of the third sample, and the background features of the fourth sample, a first loss is obtained, and based on the detection results of the first sample and / or the detection results of the second sample, a second loss is obtained.

[0059] In one implementation scenario, the first feature distance between the first sample background features and the second sample background features can be obtained, as can the second feature distance between the first sample background features and the third sample background features, and the third feature distance between the first sample background features and the fourth sample background features. Then, based on the second and / or third feature distances, and the first feature distance, a first loss is obtained. It should be noted that the first loss is positively correlated with the first feature distance, and negatively correlated with both the second and third feature distances. Therefore, the first loss characterizes the relationship between the first feature distance and the second and third feature distances. In practical applications, the first loss can be reduced by minimizing the first feature distance and maximizing the second and third feature distances, i.e., reducing the feature differences in images within a stable scene image set and maximizing the differences in images within a changing scene image set, thereby improving the detection performance of the scene detection model.

[0060] In a specific implementation scenario, the first loss can be calculated with reference to formula (3).

[0061]

[0062] Among them, L c Indicates the first loss, v1 稳 v2 represents the background features of the first sample. 稳 v1 represents the background features of the second sample. 变 v2 represents the background features of the third sample. 变 Let α represent the background features of the fourth sample, and let α represent the loss boundary.

[0063] Similarly, in situations where efficiency in calculating the first loss is crucial, variations of formula (3) can be used. For example, only one of the second and third feature distances can be considered, or the loss boundary can be disregarded, etc. See the following formula for details.

[0064]

[0065]

[0066]

[0067] The meanings of the relevant letters and symbols are completely consistent with those of the aforementioned embodiments. The above four formulas are not all the methods for calculating the first loss, and can be adapted and adjusted. No specific restrictions are imposed here.

[0068] In another implementation scenario, the second loss can be obtained based on the detection results of the first sample and / or the detection results of the second sample. That is, the second loss can be obtained based on the detection results of the first sample alone, or based on the detection results of the second sample alone, or even based on both the detection results of the first and second samples; this is not limited here.

[0069] In a specific implementation scenario, the cross-entropy loss function can be used to calculate the second loss, as shown in formula (7).

[0070]

[0071] Among them, L p Indicates the second loss. The type of the current sample image set is represented by , and z represents the sample detection result of the current sample image set. It should be noted that when obtaining the second loss based solely on the first sample detection result, z represents the first sample detection result; when obtaining the second loss based solely on the second sample detection result, z represents the second sample detection result. However, when obtaining the second loss based on both the first and second sample detection results, the first sample loss can be obtained first, followed by the second sample loss, and finally, the first and second sample losses can be weighted to obtain the second loss.

[0072] Of course, the second loss can also be calculated using methods such as mean squared error, and no specific restrictions are imposed here.

[0073] Step S504: Adjust the network parameters of the scene detection model based on the first loss and the second loss.

[0074] In one implementation scenario, the first loss and the second loss can be weighted to obtain the overall loss of the scene detection model, and the network parameters of the scene detection model can be adjusted according to the overall loss.

[0075] In a specific implementation scenario, the overall loss of the scenario detection model can be calculated by referring to formula (8).

[0076] L1=λ c ×L c +λ p ×L p +λ×J……(8)

[0077] Where L1 represents the total loss, L c Indicates the first loss, L p Indicates the second loss, λ c λ represents the weighting coefficient of the first loss. p Let λ represent the weighting coefficient of the second loss, J represent the structural risk minimization term, and λ represent the weighting coefficient of the structural risk minimization term. When initially calculating the overall loss, λ... c and λ p The parameters can be set according to empirical values, and then adjusted by referring to the overall loss calculated in the previous calculation. The adjustment is then propagated back to the scene detection model through the gradient descent algorithm, ultimately adjusting the network parameters of the scene detection model.

[0078] In another specific implementation scenario, the overall loss can be calculated based on formula (8) without considering the structural risk minimization term, thereby adjusting the network parameters of the scene detection model.

[0079] Through the aforementioned training process, on the one hand, feature extraction is performed on the images in the sample image set to obtain background features, and a first loss is obtained based on the feature distance of the background features; therefore, the first loss reflects the loss in the feature extraction process. On the other hand, prediction is performed based on the background features to obtain the sample detection result, and a second loss is obtained based on the sample detection result; therefore, the second loss reflects the loss of the output detection result. Adjusting the network parameters of the scene detection model by coordinating the first and second losses can optimize the network of the scene detection model at both the feature extraction and result detection levels, effectively improving the model training effect and ultimately enhancing the detection performance of the scene detection model.

[0080] In one implementation scenario, please refer to Figure 6 , Figure 6 yes Figure 1 A flowchart illustrating an embodiment of step S11. Specifically, step S11 in the above embodiment may include the following steps:

[0081] Step S111: Extract features from the first image and the second image in the image set respectively to obtain the first background feature of the first image and the second background feature of the second image.

[0082] In one implementation scenario, referring to the relevant description of the scene detection model training steps mentioned above (which will not be repeated here), a feature extraction network can be used to extract features from the first image and the second image in the image set, respectively, to obtain the first background feature of the first image and the second background feature of the second image.

[0083] Step S112: Perform feature comparison based on the first background feature of the first image and the second background feature of the second image in the image set to obtain the feature comparison result.

[0084] In one implementation scenario, the feature comparison result characterizes the degree of background difference between the second image and the first image in the image set. In some specific scenarios, the feature distance can be used to characterize the feature comparison result. Further, referring to the relevant description of the scene detection model training steps mentioned above, which will not be repeated here, a feature comparison network can be used to perform feature comparison based on the first background feature of the first image and the second background feature of the second image in the image set to obtain the feature comparison result.

[0085] Step S113: Based on the feature comparison results of each image set, the scene detection results are obtained.

[0086] In one implementation scenario, the feature comparison results can be directly used as the scene detection results.

[0087] In another implementation scenario, in order to intuitively see the magnitude of the scene detection result, the feature comparison result can be probability mapped. For example, a probability value can be output by the sigmoid unit and used as the scene detection result.

[0088] Therefore, by extracting features from the images in the image set, background features are obtained. Then, the background features of two images in the image set are compared to obtain scene detection results. Through feature extraction, the feature information of the images is obtained, and the image features can be expressed using abstract semantics. By comparing the features, the scene differences between the first and second images can be clearly identified using the same expression. Finally, different scene detection results can be accurately obtained based on different scene differences.

[0089] Step S12: Based on the scene detection results, at least one image set is filtered to obtain several target images, and based on the several target images, a video segment to be tested is constructed.

[0090] In one implementation scenario, a threshold can be set based on the scene detection results to filter at least one image set, resulting in several target images. It should be noted that since the scene detection result represents the probability that the second image in the image set undergoes a scene change relative to the first image, filtering the image set can yield either two images from the entire image set or just the second image from the set. Furthermore, a test video segment can be constructed using either several image sets or just the second images from several image sets.

[0091] In a specific implementation scenario, as mentioned earlier, the scene detection result is represented by the probability value z. A threshold k is set, and all images with a probability value z greater than the threshold k are selected to form the target image.

[0092] In one implementation scenario, before constructing the video segment to be tested, the selected target images can be cached in an image cache unit. Then, in response to the image cache unit being full, the video segment to be tested is constructed based on the target images currently cached in the image cache unit. Therefore, by constructing an image cache unit, it can be ensured that the number of frames in each constructed video segment to be tested meets the requirements of subsequent execution steps, which is beneficial for subsequent analysis of the video segment to be tested and improves analysis efficiency.

[0093] In a specific implementation scenario, the image buffer unit can be a single-buffered first-in-first-out (FIFO) structure. Please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of an image buffer unit with a single-buffered first-in-first-out (FIFO) structure. (Example:) Figure 7 As shown, the single-buffered FIFO structure contains only one buffer unit with a capacity of N (N is the number of image frames sent in step S13 at a time, such as 4, 8, 16, etc.). When the buffer unit is full for the first time, these N target images are used to construct the video segment to be tested. Furthermore, when the (N+1)th image frame is sent into the buffer unit, the image that was first stored in the buffer module is deleted from OUT. Therefore, the single-buffered FIFO structure can increase the real-time image data / feature reuse rate, improve the real-time performance of change detection, and improve the stability of shot pattern confirmation when the image change interval is short and the total number of change frames is small.

[0094] In another specific implementation scenario, the image caching unit can also be a double-buffered ping-pong structure. See also... Figure 8 , Figure 8 This is a schematic diagram of an image cache unit with a double-buffered ping-pong structure. (Example) Figure 8As shown, the dual-buffered ping-pong structure consists of two buffer sub-units, A and B, each with a capacity of N frames. Target images are alternately stored in buffer sub-units A and B. In the ping-pong structure, if the number of target images in either buffer sub-unit A or B is N, the target images can be used to construct a video segment to be tested. For example, when the number of target images in buffer sub-unit A is N, the N frames of target images in buffer sub-unit A can be used to construct a video segment to be tested and sent to step S13 for execution. Then, the target images are continued to be filled into buffer sub-unit B. The next time, the N frames of target images in buffer sub-unit B are used to construct a video segment to be tested and sent to step S13 for execution, and this process is repeated alternately.

[0095] Step S13: Determine the lens mode of the camera device based on the video segment to be tested.

[0096] In one implementation scenario, deep learning techniques can be used to determine the lens pattern of a camera device. Specifically, the lens pattern is detected by a pattern detection model of the video segment under test. The pattern detection model is trained on sample video segments, and these sample video segments are labeled with the lens patterns used when they were captured. Therefore, by constructing a pattern detection model using deep learning techniques, the pattern detection method can be continuously optimized with a large number of sample video segments, thereby maximizing the accuracy of lens pattern determination.

[0097] In a specific implementation scenario, such as Figure 9 As shown, Figure 9 This is a schematic diagram of the framework of an embodiment of the pattern detection model. Specifically, the pattern detection model includes a video feature extraction network and a shot pattern analysis network. The shot patterns that can be identified include: shot zooming in, arc rotation, vertical rotation, shot shaking, horizontal rotation, shot stabilization, and shot zooming out.

[0098] Further, please refer to Figure 10 , Figure 10 This is a flowchart illustrating one embodiment of the training steps for a pattern detection model. Specifically, training the pattern detection model in the above embodiment includes the following steps:

[0099] Step S101: Extract features from the sample video segments to obtain the sample video scene features of the sample video segments.

[0100] Please see Figure 11 , Figure 11 This is a schematic diagram of a sample video clip. (For example...) Figure 11 As shown, 11a is a zoom-in sample video clip, and 11b is a zoom-out sample video clip.

[0101] In one implementation scenario, a video feature extraction network is used to extract features from sample video clips to obtain sample video scene features. These sample video scene features are obtained by fusing the sample background features of each sample image within the sample video clip.

[0102] In a specific implementation scenario, the video feature extraction network can adopt an LSTM (Long Short-Term Memory) + 2D Conv (2D convolutional) structure. This involves using several sets of convolutional computation units, batch normalization units, non-linear activation layers, and pooling layers to extract background features from a single frame. The LSTM units then facilitate temporal feature fusion across multiple frames. Alternatively, this network can be replaced with common structures such as TSN (Time-Sensitive Network), CNN (Convolutional Neural Network), SlowFast model, and Video-swinTransformer model. This application does not impose specific limitations on the chosen video feature extraction network structure.

[0103] Step S102: Perform pattern classification based on the scene features of the sample video to obtain the predicted shot pattern.

[0104] In one implementation scenario, the shot pattern analysis network can classify the scene features of the sample video based on the softmax function, and finally use the one with the highest probability calculated by softmax as the predicted shot pattern of the sample video segment.

[0105] Step S103: Adjust the network parameters of the pattern detection model based on the difference between the sample shot pattern and the predicted shot pattern.

[0106] In one implementation scenario, the classification loss of the pattern detection model can be calculated based on the difference between the sample shot pattern and the predicted shot pattern, and then the network parameters of the pattern detection model can be adjusted by backpropagation through the gradient descent algorithm.

[0107] In a specific implementation scenario, the classification loss of the pattern detection model can be calculated by referring to formula (9).

[0108]

[0109] Where L2 represents the classification loss, s j Indicates the predicted shot pattern. This indicates the sample lens mode.

[0110] In the above scheme, the sample video scene features are obtained by fusing the sample background features of each sample image in the sample video clip, which fully considers the background information of each image. Then, the pattern is classified according to the sample video scene features to obtain the predicted shot pattern. Based on the difference between the predicted shot pattern and the sample shot pattern, the pattern detection model is optimized, so that the pattern detection model can significantly improve the accuracy of pattern classification through deep learning.

[0111] In one implementation scenario, please refer to Figure 12 , Figure 12 yes Figure 1 A flowchart illustrating an embodiment of step S13. Specifically, step S13 in the above embodiment may include the following steps:

[0112] Step S131: Extract features from the video segment to be tested to obtain video scene features.

[0113] In one implementation scenario, referring to the aforementioned description of the pattern detection model training steps (which will not be repeated here), a video feature extraction network can be used to extract features from the video segment under test to obtain video scene features. These video scene features are obtained by fusing the background features of each target image in the video segment under test.

[0114] In another implementation scenario, before feature extraction from the video segment under test, the segment can be normalized along the spatial and temporal dimensions. Temporal normalization aims to ensure that video segments of varying lengths maintain temporal consistency during feature extraction. Temporal normalization can be achieved using methods such as nonlinear interpolation of adjacent multi-frame images and interval sampling; this application does not impose specific limitations on these methods. Spatial normalization involves first performing linear interpolation on the temporally normalized single-frame image, and then standardizing the linearly interpolated images of uniform size (i.e., subtracting the mean and dividing by the variance).

[0115] In another implementation scenario, the pattern detection model also includes a multilayer perceptron layer, which is used to optimize the video scene features extracted by the video feature extraction network, thereby improving the utilization rate of video scene features in subsequent steps. The multilayer perceptron layer consists of several fully connected layers and nonlinear pooling layers. This application does not specify the exact number or size of the fully connected units and nonlinear pooling units.

[0116] Step S132: Perform pattern classification based on video scene features to determine the lens mode of the camera device.

[0117] In one implementation scenario, referring to the relevant description of the pattern detection model training steps mentioned above (which will not be repeated here), the lens pattern analysis network can be used to classify the video scene features into patterns to determine the lens pattern of the camera device.

[0118] In the above scheme, feature extraction is performed on the video segment under test to obtain video scene features. Then, the mode is classified according to the video scene features to determine the lens mode of the camera device. This fully utilizes the background features of the entire video segment under test for analysis and calculation, which can determine the lens mode of the camera device as accurately as possible.

[0119] In another implementation scenario, in addition to determining the lens mode of the camera device, the timing information corresponding to the current lens mode can also be determined based on the timing information of several target images that constitute the video segment to be tested.

[0120] In another implementation scenario, the timing information of the last frame of the target image constituting the video segment under test can be used as the timing information of the video segment under test, which is the timing information corresponding to the shot mode.

[0121] In the above scheme, scene detection is first performed on the image set to obtain scene detection results. Then, target images with a high probability of scene change are selected based on the scene detection results. These target images are used to construct test video segments, which are then classified to determine the camera mode. Because scene detection is performed at the image set level to obtain the probability of scene change in the image set and target images are selected accordingly, the test video segments constructed from the target images also have significant scene change characteristics. Therefore, analysis based on significant scene change characteristics can determine the lens mode of the camera device when shooting the test video segments as accurately as possible.

[0122] Please see Figure 13 , Figure 13 This is a schematic diagram of the framework of an embodiment of the lens pattern detection device 13 of this application. Specifically, the lens pattern detection device 13 includes a scene detection module 1301, a video construction module 1302, and a pattern determination module 1303. Further, the scene detection module 1301 is used to perform scene detection based on at least one image set to obtain scene detection results; wherein, the image set includes a first image and a second image captured sequentially by the camera device, and the scene detection results characterize the probability that the second image in each image set undergoes a scene change relative to the first image; the video construction module 1302 is used to filter at least one image set based on the scene detection results to obtain several target images, and construct a video segment to be tested based on the several target images; the pattern determination module 1303 is used to determine the lens pattern of the camera device based on the video segment to be tested.

[0123] In the above scheme, scene detection is first performed on the image set to obtain scene detection results. Then, target images with a high probability of scene change are selected based on the scene detection results. These target images are used to construct test video segments, which are then classified to determine the camera mode. Because scene detection is performed at the image set level to obtain the probability of scene change in the image set and target images are selected accordingly, the test video segments constructed from the target images also have significant scene change characteristics. Therefore, analysis based on significant scene change characteristics can determine the lens mode of the camera device when shooting the test video segments as accurately as possible.

[0124] In some disclosed embodiments, the scene detection module 1301 further includes a background feature extraction unit and a feature comparison unit. The background feature extraction unit is used to extract features from the first image and the second image in the image set respectively, to obtain a first background feature of the first image and a second background feature of the second image; the feature comparison unit is used to compare features based on the first background feature of the first image and the second background feature of the second image in the image set, to obtain a feature comparison result; wherein, the feature comparison result characterizes the degree of background difference between the second image and the first image in the image set; the scene detection module 1301 is used to obtain a scene detection result based on the feature comparison results of each image set.

[0125] Therefore, by extracting features from images in the image set, background features are obtained. Then, by comparing the background features of two images in the set, scene detection results are obtained. Feature extraction acquires image feature information, allowing the image features to be expressed using abstract semantics. Feature comparison then clearly identifies the scene differences between the first and second images using the same expression, ultimately leading to accurate scene detection results based on these differences.

[0126] In some disclosed embodiments, the scene detection module 1301 includes a scene detection model, and scene detection is performed by the scene detection model. The scene detection model is trained based on a first sample image set and a second sample image set. The first sample image set includes a first sample image and a second sample image, and the second sample image has no scene change compared to the first sample image. The second sample image set includes a third sample image and a fourth sample image, and the fourth sample image has a scene change compared to the third sample image.

[0127] Therefore, by building a scene detection model using deep learning technology, the scene detection method can be continuously optimized as long as a large number of sample images are provided, thereby making the scene detection results more accurate and facilitating the subsequent determination of the shot mode.

[0128] In some disclosed embodiments, the scene detection model includes a sample image acquisition subunit, which is used to select a set of images from a plurality of sample images that have not undergone scene changes to form a first sample image set; and to perform image filtering based on the plurality of sample images to obtain a second sample image set, and / or to perform image enhancement based on the first sample image set to obtain a second sample image set.

[0129] Therefore, by constructing two sets of sample images, which respectively represent scene stability and scene change, the scene detection model can perform deep learning of scene features under both conditions, effectively improving the detection performance of the scene detection model.

[0130] In some disclosed embodiments, the scene detection model includes a scene training subunit, used to extract features from a first sample image set and a second sample image set respectively, to obtain a first sample background feature of the first sample image, a second sample background feature of the second sample image, a third sample background feature of the third sample image, and a fourth sample background feature of the fourth sample image; to make predictions based on the first and second sample background features to obtain a first sample detection result, and / or to make predictions based on the third and fourth sample background features to obtain a second sample detection result; wherein, the first sample detection result represents the probability that the first sample image has a scene change relative to the second sample image, and the second sample detection result represents the probability that the third sample image has a scene change relative to the fourth sample image; to obtain a first loss based on the feature distance between the first, second, third, and fourth sample background features, and to obtain a second loss based on the first and / or second sample detection results; and to adjust the network parameters of the scene detection model based on the first and second losses.

[0131] Therefore, on the one hand, feature extraction is performed on the images in the sample image set to obtain background features, and a first loss is obtained based on the feature distance of the background features. Thus, the first loss reflects the loss in the feature extraction process. On the other hand, prediction is performed based on the background features to obtain the sample detection result, and a second loss is obtained based on the sample detection result. Thus, the second loss reflects the loss of the output detection result. Adjusting the network parameters of the scene detection model by coordinating the first and second losses can optimize the network of the scene detection model at both the feature extraction and result detection levels, effectively improving the model training effect and ultimately enhancing the detection performance of the scene detection model.

[0132] In some disclosed embodiments, the scene training subunit is further configured to obtain a first feature distance between the first sample background feature and the second sample background feature, a second feature distance between the first sample background feature and the third sample background feature, and a third feature distance between the first sample background feature and the fourth sample background feature; and to obtain a first loss based on the second feature distance and / or the third feature distance, and the first feature distance; wherein the first loss is positively correlated with the first feature distance, and the first loss is negatively correlated with both the second feature distance and the third feature distance.

[0133] Therefore, the first loss characterizes the relationship between the first feature distance and the second and third feature distances. In practical applications, the first loss can be reduced by minimizing the first feature distance and expanding the second and third feature distances, that is, reducing the feature differences of images in the stable scene image set and expanding the differences of images in the scene changing image set, thereby improving the detection effect of the scene detection model.

[0134] In some disclosed embodiments, the lens pattern detection device 13 further includes an image caching unit for caching the selected target images, and the video construction module 1302 is further configured to construct a video segment to be tested based on the target images currently cached by the image caching unit in response to the image caching unit being full.

[0135] Therefore, by constructing an image caching unit, it can be ensured that the number of frames constructed for each video segment under test can meet the requirements of subsequent execution steps, which is beneficial for the subsequent analysis of the video segment under test and improves analysis efficiency.

[0136] In some disclosed embodiments, the pattern determination module 1303 further includes a video feature extraction unit, which is used to extract features from the video segment to be tested to obtain video scene features; wherein, the video scene features are obtained by fusing the background features of each target image in the video segment to be tested; the pattern determination module 1303 is also used to perform pattern classification based on the video scene features to determine the lens mode of the camera device.

[0137] Therefore, by extracting features from the video segment under test, the video scene features are obtained, and then the mode is classified according to the video scene features to determine the lens mode of the camera device. By making full use of the background features of the entire video segment under test for analysis and calculation, the lens mode of the camera device can be determined as accurately as possible.

[0138] In some disclosed embodiments, the pattern determination module 1303 includes a pattern detection model. The shot pattern is detected by the pattern detection model of the video segment to be tested. The pattern detection model is trained based on the sample video segment, and the sample video segment is labeled with the sample shot pattern when the sample video segment was shot.

[0139] Therefore, by constructing a pattern detection model using deep learning technology, the pattern detection method can be continuously optimized by providing a large number of sample video clips, thereby improving the accuracy of determining shot patterns as much as possible.

[0140] In some disclosed embodiments, the pattern detection model further includes a pattern training subunit for extracting features from sample video segments to obtain sample video scene features of the sample video segments; wherein, the sample video scene features are obtained by fusing sample background features of each sample image in the sample video segment; pattern classification is performed based on the sample video scene features to obtain predicted shot patterns; and the network parameters of the pattern detection model are adjusted based on the difference between the sample shot patterns and the predicted shot patterns.

[0141] Therefore, by fusing the background features of each sample image in the sample video clip, the scene features of the sample video are obtained, fully considering the background information of each image. Then, pattern classification is performed based on the scene features of the sample video to obtain the predicted shot pattern. Based on the difference between the predicted shot pattern and the sample shot pattern, the pattern detection model is optimized, so that the pattern detection model can significantly improve the accuracy of pattern classification through deep learning.

[0142] Please see Figure 14 , Figure 14 This is a schematic diagram of a framework of an embodiment of the electronic device 14 of this application. Specifically, the electronic device 14 includes a processor 1401 and a memory 1402. The memory 1402 is coupled to the processor 1401. The processor 1401 is used to execute program instructions stored in the memory 1402 to implement the steps of any embodiment of the lens pattern detection method.

[0143] Specifically, processor 1401 can also be referred to as a CPU (Central Processing Unit). Processor 1401 may be an integrated circuit chip with signal processing capabilities. Processor 1401 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 1401 can be implemented by multiple integrated circuit chips.

[0144] In the above scheme, scene detection is first performed on the image set to obtain scene detection results. Then, target images with a high probability of scene change are selected based on the scene detection results. These target images are used to construct test video segments, which are then classified to determine the camera mode. Because scene detection is performed at the image set level to obtain the probability of scene change in the image set and target images are selected accordingly, the test video segments constructed from the target images also have significant scene change characteristics. Therefore, analysis based on significant scene change characteristics can determine the lens mode of the camera device when shooting the test video segments as accurately as possible.

[0145] Please see Figure 15 , Figure 15 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium 15 of this application. In this embodiment, the computer-readable storage medium 15 stores processor-executable program instructions 1501, which are used to perform the steps in the above-described target region prediction method embodiment.

[0146] The computer-readable storage medium 15 can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or a medium that can store program instructions. Alternatively, it can be a server that stores the program instructions, which can send the stored program instructions to other devices for execution or execute the stored program instructions itself.

[0147] In the above scheme, scene detection is first performed on the image set to obtain scene detection results. Then, target images with a high probability of scene change are selected based on the scene detection results. These target images are used to construct test video segments, which are then classified to determine the camera mode. Because scene detection is performed at the image set level to obtain the probability of scene change in the image set and target images are selected accordingly, the test video segments constructed from the target images also have significant scene change characteristics. Therefore, analysis based on significant scene change characteristics can determine the lens mode of the camera device when shooting the test video segments as accurately as possible.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A lens pattern detection method, characterized in that, include: Scene detection is performed based on at least one image set to obtain scene detection results; wherein, the image set includes a first image and a second image captured sequentially by a camera device, and the scene detection results characterize the probability that the second image in each of the image sets undergoes a scene change relative to the first image; Based on the scene detection results, the at least one image set is filtered to obtain several target images, and a video segment to be tested is constructed based on the several target images; Based on the video segment to be tested, determine the lens mode of the camera device; Determining the lens mode of the camera device based on the video segment to be tested includes: Feature extraction is performed on the video segment to be tested to obtain video scene features; wherein, the video scene features are obtained by fusing the background features of each target image in the video segment to be tested; Based on the video scene features, the camera device's lens mode is determined by pattern classification. The lens mode includes: zooming in, arc rotation, vertical rotation, lens shaking, horizontal rotation, lens stabilization, and / or zooming out.

2. The method according to claim 1, characterized in that, The process of performing scene detection based on several image sets to obtain scene detection results includes: Feature extraction is performed on the first image and the second image in the image set respectively to obtain the first background feature of the first image and the second background feature of the second image; Based on the first background feature of the first image and the second background feature of the second image in the image set, a feature comparison is performed to obtain a feature comparison result; wherein, the feature comparison result characterizes the degree of background difference between the second image and the first image in the image set; The scene detection result is obtained based on the feature comparison results of each image set.

3. The method according to claim 1 or 2, characterized in that, The scene detection is performed by a scene detection model, which is trained based on a first sample image set and a second sample image set. The first sample image set includes a first sample image and a second sample image, and the second sample image does not have a scene change compared to the first sample image. The second sample image set includes a third sample image and a fourth sample image, and the fourth sample image has a scene change compared to the third sample image.

4. The method according to claim 3, characterized in that, The steps for obtaining the first sample image set and the second sample image set include: The first sample image set is formed by selecting a set of images from a number of sample images that have not undergone scene changes. Image filtering is performed based on the aforementioned sample images to obtain the second sample image set, and / or image enhancement is performed based on the first sample image set to obtain the second sample image set.

5. The method according to claim 3, characterized in that, The training steps for the scene detection model include: Feature extraction is performed on the first sample image set and the second sample image set respectively to obtain the first sample background feature of the first sample image, the second sample background feature of the second sample image, the third sample background feature of the third sample image, and the fourth sample background feature of the fourth sample image; A first sample detection result is obtained by predicting based on the first sample background features and the second sample background features, and / or a second sample detection result is obtained by predicting based on the third sample background features and the fourth sample background features; wherein, the first sample detection result represents the probability that the first sample image has a scene change relative to the second sample image, and the second sample detection result represents the probability that the third sample image has a scene change relative to the fourth sample image; A first loss is obtained based on the feature distance between the first sample background features, the second sample background features, the third sample background features, and the fourth sample background features, and a second loss is obtained based on the first sample detection result and / or the second sample detection result; Based on the first loss and the second loss, the network parameters of the scene detection model are adjusted.

6. The method according to claim 5, characterized in that, The first loss is obtained based on the feature distances between the first sample background features, the second sample background features, the third sample background features, and the fourth sample background features, including: Obtain the first feature distance between the background features of the first sample and the background features of the second sample, obtain the second feature distance between the background features of the first sample and the background features of the third sample, and obtain the third feature distance between the background features of the first sample and the background features of the fourth sample; The first loss is obtained based on the second feature distance and / or the third feature distance, and the first feature distance; wherein the first loss is positively correlated with the first feature distance, and the first loss is negatively correlated with both the second feature distance and the third feature distance.

7. The method according to claim 1, characterized in that, Before constructing the video segment to be tested based on the plurality of target images, the method further includes: The selected target images are cached in the image cache unit; The process of constructing the video segment to be tested based on the plurality of target images includes: In response to the image cache unit being full, the video segment to be tested is constructed based on the target image currently cached in the image cache unit.

8. The method according to claim 1, characterized in that, The shot pattern is obtained by a pattern detection model from the video segment to be tested. The pattern detection model is trained based on sample video segments, and the sample video segments are labeled with the sample shot patterns when the sample video segments were filmed.

9. The method according to claim 8, characterized in that, The training steps of the pattern detection model include: Feature extraction is performed on the sample video segment to obtain the sample video scene features of the sample video segment; wherein, the sample video scene features are obtained by fusing the sample background features of each sample image in the sample video segment; Based on the scene features of the sample video, pattern classification is performed to obtain the predicted shot pattern; Based on the difference between the sample shot pattern and the predicted shot pattern, the network parameters of the pattern detection model are adjusted.

10. A lens pattern detection device, characterized in that, include: A scene detection module is used to perform scene detection based on at least one image set to obtain scene detection results; wherein, the image set includes a first image and a second image captured sequentially by a camera device, and the scene detection results characterize the probability that the second image in each image set undergoes a scene change relative to the first image; The video construction module is used to filter the at least one image set based on the scene detection results to obtain several target images, and to construct a video segment to be tested based on the several target images; The mode determination module is used to determine the lens mode of the camera device based on the video segment to be tested; The pattern determination module is further configured to extract features from the video segment to be tested to obtain video scene features; wherein, the video scene features are obtained by fusing the background features of each target image in the video segment to be tested; Based on the video scene features, the camera device's lens mode is determined by pattern classification. The lens mode includes: zooming in, arc rotation, vertical rotation, lens shaking, horizontal rotation, lens stabilization, and / or zooming out.

11. An electronic device, characterized in that, The device includes a processor and a memory, the memory being coupled to the processor, the processor being used to execute program instructions stored in the memory to implement the lens pattern detection method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the lens pattern detection method according to any one of claims 1-9.

Images