Camera shift identification method and system based on twin network, terminal and medium

Abstract

The invention provides a camera shift identification method and system based on a twin network. The method comprises the following steps: acquiring a front and back frame image data set containing multiple scenes; using a twin network model to perform feature extraction on the front and back frame image data to obtain front and back frame multilayer features; performing feature difference calculation on the multi-layer features of the front and back frames by using a space attention mechanism to obtain multi-layer difference features; carrying out difference feature fusion and classification on the multi-layer difference features, and training to obtain a multi-task model combining a twin network and a space attention mechanism; and optimizing the multi-task model to obtain a camera shift detection model for performing camera shift detection on the to-be-identified image. According to the method, the twin network and the space attention mechanism are combined, so that the feature comparison precision and the shift detection sensitivity are improved. A space attention mechanism is introduced and is used for dynamically paying attention to key region differences of front and back frames, and the perception ability of the model to local saliency features is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of camera displacement detection, and in particular to a camera displacement recognition method, system, terminal and medium based on a twin network. Background Art

[0002] With the popularity of modern video surveillance equipment, cameras are increasingly deployed. In some key scenarios, such as safe production and sensitive areas, changes in camera position may violate relevant regulations or directly affect monitoring results, or even cause safety hazards. This position change may be caused by human adjustment, external environmental interference, or equipment abnormality. In addition, changes in camera position due to changes in the external environment may also cause monitoring failure. Therefore, accurately identifying whether the camera has shifted has become a key technical problem.

[0003] At present, most methods for identifying camera shift rely on manual inspection or traditional computer vision technology, such as the camera shift detection method based on image matching. This method extracts the feature points of the front and rear frames of the camera and uses image registration technology to compare their spatial distribution and differences to determine whether the camera has shifted. This process usually includes the following steps: first, extract significant feature points from the image, such as corners or edge features; then use a specific registration algorithm (such as RANSAC) to match and align the feature points of the front and rear frames; finally, determine whether the camera has changed position by analyzing the distribution change or offset of the feature points. However, this camera shift detection method based on image matching usually has the following technical problems:

[0004] It is highly dependent on the environment and cannot adapt to complex scenes such as lighting changes, dynamic occlusion, and camera screen distortion.

[0005] Lacking end-to-end deep learning model support, recognition results are limited by manual feature design, and it is difficult to effectively process low-quality or damaged image inputs.

[0006] When the camera has extreme conditions such as screen distortion or blur, misjudgment is likely to occur.

[0007] There are certain bottlenecks in real-time performance, especially when processing high-resolution videos, which places high demands on CPU and storage resources. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the present invention provides a camera shift recognition method, system, terminal and medium based on a twin network.

[0009] According to one aspect of the present invention, a camera shift recognition method based on a twin network is provided, comprising:

[0010] Obtain a dataset of front and back frame images containing various scenes;

[0011] A twin network model is used to extract features from the image data of the previous and next frames to obtain multi-layer features of the previous and next frames; a spatial attention mechanism is used to calculate feature differences of the multi-layer features of the previous and next frames to obtain multi-layer difference features; the multi-layer difference features are fused and classified to obtain a multi-task model combining the twin network and the spatial attention mechanism through training;

[0012] The multi-task model is optimized to obtain a camera shift detection model for performing camera shift recognition on an image to be recognized.

[0013] Preferably, the above method further comprises:

[0014] The camera shift detection model is used to perform camera shift recognition on the image to be recognized.

[0015] According to another aspect of the present invention, a camera shift recognition system based on a twin network is provided, comprising:

[0016] A data acquisition module, which is used to acquire a dataset of front and back frame images containing a variety of scenes;

[0017] A model training module, which uses a twin network model to extract features from the image data of the previous and next frames to obtain multi-layer features of the previous and next frames; uses a spatial attention mechanism to calculate feature differences on the multi-layer features of the previous and next frames to obtain multi-layer difference features; performs difference feature fusion and classification on the multi-layer difference features to train a multi-task model combining the twin network and the spatial attention mechanism;

[0018] A model optimization module is used to optimize the multi-task model to obtain a camera shift detection model for performing camera shift recognition on the image to be recognized.

[0019] Preferably, the above system further includes:

[0020] A shift recognition module uses the camera shift detection model to perform camera shift recognition on the image to be recognized.

[0021] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can be used to execute the method described above in the present invention, or to run the system described above in the present invention.

[0022] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute the method described above in the present invention, or to run the system described above in the present invention.

[0023] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0024] The camera shift recognition method, system, terminal and medium based on the twin network provided by the present invention combine the twin network and the spatial attention mechanism to build a multi-task model, which improves the accuracy and robustness of camera shift recognition, captures subtle changes through deep network and multi-level feature modeling, and improves the accuracy of feature comparison and the sensitivity of shift detection.

[0025] The camera shift recognition method, system, terminal and medium based on the twin network provided by the present invention accurately distinguish between three types of situations: shift, non-shift and rejection judgment through a multi-task learning framework, thereby achieving adaptation to complex scenes (such as lighting changes, dynamic occlusion, screen noise, etc.), especially effectively dealing with extreme situations through a rejection judgment mechanism.

[0026] The camera shift recognition method, system, terminal and medium based on the twin network provided by the present invention can run efficiently on edge computing devices, realize real-time processing, and support real-time processing; make full use of the AI ​​computing power of edge computing devices, optimize computing resource requirements, and meet the high requirements of safe production and sensitive area monitoring.

[0027] The camera shift recognition method, system, terminal and medium based on the twin network provided by the present invention dynamically focus on the differences between the key areas of the previous and next frames through the application of the spatial attention mechanism in feature comparison, thereby enhancing the model's perception of local significant features, which not only improves the robustness and recognition accuracy, but also significantly reduces the misjudgment and missed judgment rates, and expands the actual application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0029] Figure 1 This is a workflow diagram of a camera shift recognition method based on a twin network in a preferred embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of the component modules of a camera shift recognition system based on a twin network in a preferred embodiment of the present invention.

[0031] Figure 3This is a working architecture diagram of a multi-task model that combines a twin network and a spatial attention mechanism in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0033] In response to the problems existing in the prior art such as poor adaptability to complex scenes and insufficient differentiation ability, especially problems in complex scenes such as lighting changes, dynamic occlusion and possible low-quality input from the camera, an embodiment of the present invention provides a camera shift recognition method based on a twin network, which uses a twin network combined with a spatial attention mechanism to realize camera shift detection.

[0034] Specifically, Figure 1 As shown, the camera shift recognition method based on the twin network provided in this embodiment may include:

[0035] S1, obtain a dataset of front and back frame images containing various scenes;

[0036] S2, using a twin network model to extract features from the image data of the previous and next frames, and obtain multi-layer features of the previous and next frames; using the spatial attention mechanism to calculate the feature differences of the multi-layer features of the previous and next frames, and obtain multi-layer difference features; performing difference feature fusion and classification on the multi-layer difference features, and training to obtain a multi-task model combining the twin network and the spatial attention mechanism;

[0037] S3, optimizing the multi-task model to obtain a camera shift detection model for performing camera shift recognition on the image to be recognized.

[0038] In some preferred embodiments, the above method may further include the following operations:

[0039] S4, using the camera shift detection model to perform camera shift recognition on the image to be recognized.

[0040] In some preferred implementations, the above S1, obtaining a dataset of previous and next frame images containing multiple scenes, may further include the following operations:

[0041] S11, collects a large amount of front and back frame image data, including front and back frame image data in various scenarios such as lighting changes, dynamic occlusion, and camera screen blur and other extreme scenarios;

[0042] S12, performing a variety of reasonable enhancement operations on the image data of the previous and next frames, including: illumination adjustment and affine transformation, edge filling and aspect ratio adjustment.

[0043] In some preferred embodiments, the above S2, using a twin network model to extract features from the image data of the previous and next frames to obtain multi-layer features of the previous and next frames, may further include the following operations:

[0044] A deep residual network model based on a twin network architecture is provided to extract multi-level features from the input front and back frame image data respectively, to obtain shallow features for retaining more spatial information, middle-level features containing object contours, shapes and local semantic information, and global semantic deep features containing the overall changes of the scene and key targets.

[0045] In some preferred implementations, the above S2, using the spatial attention mechanism to calculate the feature difference of the multi-layer features of the previous and next frames respectively to obtain the multi-layer difference features, may further include the following operations:

[0046] S21, flattening the extracted multi-level features into one-dimensional feature vectors respectively;

[0047] S22, for each level of feature one-dimensional feature vector, respectively calculate their absolute difference between the previous and next frames, and obtain multi-layer feature differences used to characterize the spatial changes of the previous and next frames at different levels; among them, the shallow feature difference is used to capture details and local position changes, the middle feature difference is used to characterize the difference in shape and contour, and the deep feature difference is used to reflect the overall semantic change;

[0048] S23, uses the spatial attention mechanism to weight the multi-layer feature differences and obtains the multi-layer key area features caused by camera shift;

[0049] S24, scale alignment and processing of the multi-layer key area features are performed to obtain multi-layer difference features.

[0050] In some preferred implementations, the above S23 uses a spatial attention mechanism to weight the multi-layer feature differences to obtain multi-layer key area features caused by camera shift, and may further include the following operations:

[0051] S231, through the first layer of convolution, channel compression is performed on multi-layer feature differences to extract low-dimensional features and generate preliminary key area representation;

[0052] S232, using the preliminary key area representation, a single-channel attention map is generated through the second layer of convolution. The single-channel attention map normalizes the weight distribution of the features at each position in the multi-layer feature difference through the Sigmoid activation function to form a spatial weight distribution;

[0053] S233, applying the attention map to the corresponding feature differences by point-by-point weighting to highlight the features of the key areas and obtain multi-layer key area features.

[0054] In some preferred embodiments, in the above S23, the preliminary key area representation generated by the first layer of convolution is used as an intermediate result to generate a single-channel attention map, which has the following functions: (1) Preliminary channel compression of multi-layer feature differences, extraction of low-dimensional key area representation, removal of redundant information, and enhancement of the ability to distinguish key areas. This step ensures that the input feature dimension is lower and the structure is simpler, thereby providing a more efficient feature representation for subsequent attention map generation. (2) If a single-channel attention map is generated directly, the original multi-layer feature differences may be too complex and contain redundant or noisy information, resulting in a decrease in the quality of the attention map. This step uses two layers of convolution. First, through the compression of the first layer of convolution, it can focus on the key area more efficiently in the low-dimensional space, remove redundant information, and improve the representation ability of the key area. Through two layers of convolution, the first layer reduces the dimension after compression, and the second layer is processed to generate the attention map, which can effectively reduce the amount of calculation.

[0055] In a specific application example, in the above S23, the dimension of the obtained multi-layer feature difference (original feature map) is [1,64,56,56], that is, nchw, where n refers to the number of training batches, c refers to the number of feature map channels, h refers to the width of the feature map, and w refers to the width of the feature map; after the first layer of convolution, the dimension of the feature map becomes [1,32,56,56] (channel number compression, where the number of channels is halved); after the second layer of convolution, a single-channel attention map is obtained, that is, the dimension of the feature map becomes [1,1,56,56]. After the Sigmoid function is activated, the output dimension is still [1,1,56,56], but the value of each position element changes to between 0 and 1, and the features of each position in the obtained multi-layer feature difference are each position in the 56*56 (corresponding to the width and height of the feature map) area of ​​the original feature map.

[0056] In some preferred implementations, the above S24, performing scale alignment and processing on the multiple layers of key area features to obtain the multiple layers of difference features, may further include the following operations:

[0057] S241, unifies the spatial structure of multi-layer key region features through upsampling;

[0058] S242, for the key area features of each layer, extract the difference area features therein by superimposing convolution and activation functions;

[0059] S243, the mean operation is used to pool the difference area features in the spatial dimension to obtain multi-layer difference features.

[0060] In some preferred implementations, the above S2, performing differential feature fusion and classification on multiple layers of differential features to obtain a multi-task model combining a twin network and a spatial attention mechanism, may further include the following operations:

[0061] S2i, integrates the difference features of shallow, middle and deep layers into multi-scale feature representation through channel splicing operation, and obtains the fused multi-scale features that combine the spatial and semantic information of each level;

[0062] S2ii, the fused multi-scale features are input into the classification head for judgment, and a multi-task model combining the twin network and the spatial attention mechanism is obtained.

[0063] In some preferred embodiments, the above S2ii, the classification head, includes: a two-layer fully connected network; wherein:

[0064] The first layer of the fully connected network uses the ReLU activation function to enhance the nonlinear expression ability of features and distinguish between shifted and non-shifted patterns in the high-dimensional feature space;

[0065] The second layer of the fully connected network performs the classification tasks of shift, non-shift and rejection through the Softmax classifier.

[0066] In some preferred implementations, the above S3, optimizing the multi-task model, may further include the following operations:

[0067] The cross entropy loss function is used in combination with the contrast loss function to optimize the multi-task model;

[0068] Cross entropy loss function, used to optimize the classification task of the multi-task model;

[0069] The contrast loss function calculates the Euclidean distance between the features of the previous and next frames and imposes weighted constraints on the shift, non-shift, and rejection category labels, so that the multi-task model can effectively distinguish the shift features in different scenarios.

[0070] In some preferred implementations, the above contrast loss constraint may further include:

[0071] For the screened shift and rejection samples, the multi-task model regards the shift categories and rejection categories as shift categories during the contrast loss calculation process, and calculates the feature distance in the embedding space through the contrast loss function to dynamically adjust the feature similarity.

[0072] In some preferred implementations, the above optimization of the multi-task model may further include the following operations:

[0073] A controllable weight parameter is introduced into the cross entropy loss and contrast loss to balance the cross entropy loss and the contrast loss, so that the multi-task model can cope with the complexity of data distribution and the weight distribution problems between tasks while optimizing the shift recognition ability.

[0074] Based on the same inventive concept, an embodiment of the present invention also provides a camera shift recognition system based on a twin network.

[0075] Specifically, Figure 2 As shown, the camera shift recognition system based on the twin network provided in this embodiment may include:

[0076] A data acquisition module, which is used to acquire a dataset of front and back frame images containing a variety of scenes;

[0077] Model training module: This module uses a twin network model to extract features from the image data of the previous and next frames to obtain multi-layer features of the previous and next frames; uses the spatial attention mechanism to calculate the feature differences of the multi-layer features of the previous and next frames to obtain multi-layer difference features; performs difference feature fusion and classification on the multi-layer difference features, and trains a multi-task model that combines the twin network and the spatial attention mechanism;

[0078] A model optimization module is used to optimize the multi-task model to obtain a camera shift detection model for camera shift recognition of the image to be recognized.

[0079] In some preferred embodiments, the above system may further include:

[0080] The shift recognition module uses the camera shift detection model to perform camera shift recognition on the image to be recognized.

[0081] The working content of the camera shift recognition system based on the twin network provided in the above embodiment of the present invention is further described in detail below.

[0082] The camera shift recognition system provided in this embodiment uses a twin network combined with a spatial attention mechanism to implement camera shift detection. In the model training stage, a large amount of previous and next frame image data is first collected to cover extreme scenes such as lighting changes, dynamic occlusion, and possible screen distortion and blurring of the camera, and a deep learning model is used for feature extraction and comparative analysis.

[0083] In addition, in order to enhance the generalization performance of the model, the model performs a variety of reasonable enhancement operations on the data during the training phase, including illumination adjustment and affine transformation. These operations are mainly aimed at brightness changes and image geometric distortions that may occur in actual camera monitoring scenes. Common enhancement operations such as random cropping and rotation are not applicable to camera shift detection tasks because they may destroy spatial consistency, thereby affecting the model's accurate capture of shift features. Therefore, more suitable enhancement methods such as edge padding and aspect ratio adjustment are selected to improve data diversity while maintaining spatial consistency, and better meet the needs of camera shift judgment.

[0084] like Figure 3 As shown in the figure, it is a working architecture diagram of a multi-task model combining a twin network and a spatial attention mechanism, where the model design includes the following steps:

[0085] Step 1: Deep feature extraction

[0086] ResNet18 based on the twin network architecture is used to extract multi-level features from the input front and back frame images. The structure of the twin network contains two parallel branch networks that share parameters and are used to extract features from the input front and back frame images.

[0087] First, the input image is passed through the initial feature extraction module of ResNet18 to generate shallow features. These features retain more spatial information and help to accurately capture the position changes caused by camera shift.

[0088] Next, the image passes through the middle layer to generate middle-level features containing object contours, shapes, and local semantic information. These features are used to capture larger difference areas between previous and next frames.

[0089] Finally, the image passes through a deep feature module to extract global semantic information, including the overall changes of the scene and the salient features of key targets, which is particularly important for camera shift judgment in complex backgrounds.

[0090] Through the design of the twin network, features at different levels are organized into a unified multi-scale representation, and play different roles in camera shift recognition. This multi-level feature modeling can capture the feature differences caused by camera shift from local to global. In addition, for dynamic scenes and extreme situations such as screen blur and blur that may occur in the camera, the stability of deep semantic information is used to effectively reduce the impact of external interference.

[0091] Step 2: Difference feature extraction

[0092] In difference feature extraction, the difference between the features of the previous and next frames is modeled and analyzed through the following steps:

[0093] (1) Feature difference calculation

[0094] Flatten multi-layer features: Flatten the extracted multi-layer features into a one-dimensional feature vector to ensure that the features of each frame are aligned and compared in the same dimension.

[0095] Difference calculation: For shallow, middle and deep features, the absolute difference between the previous and next frames is calculated respectively, and the spatial changes of the previous and next frames are modeled from different levels. The shallow feature difference is used to capture details and local position changes, the middle feature difference mainly represents the difference in shape and contour, and the deep feature difference reflects the overall semantic change. This hierarchical modeling helps to accurately distinguish the displacement characteristics of the camera.

[0096] (2) Enhanced attention to feature differences

[0097] The spatial attention mechanism is used to weight the feature differences. Its core function is to dynamically focus on the most significant spatial change areas in the features of the previous and next frames, thereby highlighting the key area features caused by camera shift.

[0098] First, the first layer of convolution is used to compress the feature differences, extract low-dimensional features, and generate a preliminary key area representation. This process can effectively reduce redundant information and focus on areas with significant differences.

[0099] Next, based on the preliminary key area representation, the second layer of convolution generates a single-channel attention map, which uses the Sigmoid activation function to normalize the weights of the features at each position in the multi-layer feature differences to form a spatial weight distribution.

[0100] Finally, the attention map is applied to the feature differences by weighting point by point, making the features of the key areas more prominent while the influence of the non-key areas is weakened. This mechanism significantly improves the ability to capture local salient features and provides more discriminative feature input for subsequent classification.

[0101] (3) Scale alignment and processing of feature differences

[0102] The feature differences are restored to a uniform spatial resolution through upsampling, which is the basis for joint modeling of cross-level features. Upsampling ensures that the spatial structures of features at different levels are consistent, which facilitates subsequent splicing and fusion. The feature differences of each layer are further extracted through a dedicated convolution module, which deeply mines the expressive power of features by superimposing convolutions and activation functions, thereby enhancing the representation of difference areas. The features are pooled using the mean operation in the spatial dimension, retaining global difference information while significantly reducing computational overhead. This pooling operation can also compress redundant information and strengthen the focus on key features. The design of the scale alignment and processing module ensures the consistency of shallow, middle and deep features in spatial resolution and expressive power, providing high-quality input for subsequent feature fusion and classification.

[0103] Through the above (1) to (3), multi-layer difference features are obtained.

[0104] (4) Differential feature fusion and classification

[0105] In the process of differential feature fusion and classification, the shallow, middle and deep differential features are first integrated into a multi-scale feature representation through channel splicing operations. This hierarchical fusion design fully combines the spatial and semantic information of each layer and effectively improves the representation ability of the features. The fused multi-scale features are input into the classification head for judgment.

[0106] The classification head consists of two layers of fully connected networks. The first layer uses the ReLU activation function to enhance the nonlinear expression ability of the features, ensuring better distinction between shifted and non-shifted patterns in the high-dimensional feature space. The second layer uses the Softmax classifier to complete three classification tasks: shifted, non-shifted, and rejected. Among them, the rejection judgment is designed to deal with extreme situations such as camera screen distortion and blur. When the quality of the input data is not enough to provide reliable classification, the system will output a rejection judgment result. This design effectively avoids misjudgment and overconfident classification results, and provides higher reliability for complex scenarios in practical applications.

[0107] The design of the classification head combines contextual information and multi-level features, making the classification results more stable and accurate in complex scenes. In addition, the multi-scale feature fusion strategy can cope with subtle differences and complex background changes in camera shift detection, effectively improving the robustness and generalization ability of the model.

[0108] Step 3: Model optimization

[0109] The optimization process uses the cross entropy loss function for multi-task classification optimization, combined with contrast loss constraints to improve the model's sensitivity to shift detection. Contrastive loss calculates the Euclidean distance between the features of the previous and next frames and imposes weighted constraints on the category labels (shift or non-shift) to ensure that the model can effectively distinguish shift features in different scenarios. Specifically, for the selected shift and rejection samples, the model treats category 1 (shift) and category 2 (rejection) as shift categories during the contrast loss calculation process, and calculates the feature distance in the embedding space through the contrast loss function to dynamically adjust feature similarity. A controllable weight parameter is introduced to balance the classification loss and contrast loss, so that the model can effectively deal with the complexity of data distribution and the weight distribution problems between tasks while optimizing the shift recognition ability.

[0110] Through the above steps 1 to 3, the camera shift detection model is trained and optimized. Using the obtained camera shift detection model, the image to be detected is used as the model input, and the classification result is output to complete the camera shift recognition.

[0111] It should be noted that the steps in the method provided by the present invention can be implemented by using the corresponding components in the system, and those skilled in the art can refer to the technical solution of the system to implement the step flow of the method, and can also refer to the technical solution of the method to implement the composition of the system, that is, the embodiments in the system and the embodiments in the method can be understood as preferred examples of each other, which will not be elaborated here.

[0112] The camera shift recognition method and system provided by the above embodiments of the present invention combine the twin network and the spatial attention mechanism to realize the multi-task model design, thereby improving the accuracy of feature comparison and the sensitivity of shift detection. The spatial attention mechanism is introduced in the feature comparison to dynamically focus on the differences in the key areas of the previous and next frames, thereby enhancing the model's perception of local significant features.

[0113] An embodiment of the present invention further provides a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor can be used to execute any method of the above-mentioned embodiments of the present invention, or to run any system of the above-mentioned embodiments of the present invention.

[0114] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0115] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0116] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0117] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0118] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0119] The camera shift recognition method, system, terminal and medium based on the twin network provided by the above embodiments of the present invention improve the accuracy and robustness of camera shift recognition, capture subtle changes through deep network and multi-level feature modeling; achieve adaptation to complex scenes (such as lighting changes, dynamic occlusion, screen distortion, etc.), especially through the rejection judgment mechanism to effectively deal with extreme situations; support real-time processing, can make full use of the AI ​​computing power of edge computing devices, optimize computing resource requirements, and meet actual application needs.

[0120] The camera shift recognition method, system, terminal and medium based on the twin network provided by the above-mentioned embodiments of the present invention, combined with the multi-task model of the twin network and the spatial attention mechanism, perform deep learning modeling on multi-level image features through the feature extraction module, and use the spatial attention mechanism to enhance attention to the difference features of key areas. While realizing camera shift detection, the model accurately distinguishes three types of situations: shift, non-shift and rejection judgment through a multi-task learning framework. In addition, current edge computing devices usually have a certain AI computing power. This model can run efficiently on edge computing devices to achieve real-time processing, meeting the high requirements of safe production and sensitive area monitoring. This method not only improves the robustness and recognition accuracy, but also significantly reduces the misjudgment and missed judgment rates, and expands the practical application scenarios of the model.

[0121] The camera shift recognition method, system, terminal and medium based on the twin network provided by the above-mentioned embodiment of the present invention significantly improve the accuracy of detection through the multi-level modeling capability and attention mechanism of the deep network, and make full use of the multi-scale fusion of shallow, middle and deep features to ensure excellent performance in capturing subtle differences in camera shift. At the same time, by introducing a rejection judgment mechanism, it can effectively deal with extreme situations such as camera screen distortion and blur, avoid misjudgment and improve the reliability of classification. In addition, the optimized model structure and reasonable data enhancement strategy enable it to adapt to complex and dynamic monitoring scenarios. The end-to-end deep learning model can not only fully mine multi-level features, but also use the AI ​​computing power on the edge computing device to optimize the computing resource requirements through quantization, model pruning and distillation technology, thereby significantly reducing the computing cost. This design enables the model to achieve efficient and real-time camera shift detection in complex scenarios, meeting the actual application needs while improving the overall reliability and deployment flexibility of the system.

[0122] All matters not covered in the above embodiments of the present invention are well known in the art.

[0123] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A camera displacement recognition method based on a twin network, characterized in that: include: Obtain a dataset of front and back frame images containing various scenes; Using a twin network model to extract features from the front and back frame image data to obtain multi-layer features of the front and back frames; Using a spatial attention mechanism, feature differences are calculated for the multi-layer features of the previous and next frames to obtain multi-layer difference features; Performing differential feature fusion and classification on the multi-layer differential features, and training to obtain a multi-task model combining a twin network and a spatial attention mechanism; The multi-task model is optimized to obtain a camera shift detection model for performing camera shift recognition on an image to be recognized.

2. The camera displacement recognition method based on twin network according to claim 1 is characterized in that: The obtaining of a dataset of front and back frame images containing a variety of scenes includes: Collect multiple front and back frame image data, including front and back frame image data in various scenes such as lighting changes, dynamic occlusion, and camera screen distortion and blur.

3. The camera displacement recognition method based on twin network according to claim 2 is characterized in that: Also includes: A variety of enhancement operations are performed on the preceding and following frame image data, including: illumination adjustment and affine transformation as well as edge filling and aspect ratio adjustment.

4. The camera displacement recognition method based on twin network according to claim 1 is characterized in that: Also includes any one or more of the following: - The use of a twin network model to extract features from the front and back frame image data to obtain multi-layer features of the front and back frames includes: A deep residual network model based on a twin network architecture is provided to extract multi-level features from the input front and back frame image data to obtain corresponding shallow features, middle features and deep features; wherein: The shallow features are used to reflect spatial information; The middle-level features are used to reflect the object's contour, shape and local semantic information; The deep features are used to reflect the global semantic information of overall changes and key targets; - The spatial attention mechanism is used to calculate the feature difference of the multi-layer features of the previous and next frames to obtain multi-layer difference features, including: Flattening the extracted multi-level features into one-dimensional feature vectors respectively; For each level of feature, the absolute difference between the previous and next frames is calculated to obtain multi-layer feature differences that are used to characterize the spatial changes of the previous and next frames at different levels. The shallow feature differences are used to capture details and local position changes, the middle feature differences are used to characterize the differences in shape and contour, and the deep feature differences are used to reflect the overall semantic changes. The multi-layer feature differences are weighted using a spatial attention mechanism to obtain multi-layer key area features caused by camera shift; Scale-aligning and processing the multi-layer key area features to obtain multi-layer difference features; - The multi-layer difference features are fused and classified to obtain a multi-task model combining a twin network and a spatial attention mechanism, including: Through channel splicing operations, the difference features of shallow, middle and deep layers are integrated into multi-scale feature representation, and the fused multi-scale features that combine the spatial and semantic information of each layer are obtained; The fused multi-scale features are input into the classification head for judgment, and a multi-task model combining the twin network and the spatial attention mechanism is obtained.

5. The camera displacement recognition method based on twin network according to claim 4 is characterized in that: Also includes any one or more of the following: -The spatial attention mechanism is used to weight the multi-layer feature differences to obtain multi-layer key area features caused by camera shift, including: Perform channel compression on the multi-layer feature differences through the first convolution layer, extract low-dimensional features, and generate a preliminary key area representation; Using the preliminary key area representation, a single-channel attention map is generated through a second layer of convolution, and the single-channel attention map performs normalized weight allocation on the features of each position in the multi-layer feature difference through a Sigmoid activation function to form a spatial weight distribution; Applying the attention map to the corresponding multi-layer feature differences by point-by-point weighting to highlight the features of the key area and obtain multi-layer key area features; - The scale alignment and processing of the multi-layer key area features to obtain multi-layer difference features includes: Unifying the spatial structures of the multiple layers of key area features by upsampling; For each layer of unified key area features, the difference area features are extracted by superimposing convolution and activation functions; The difference region features are pooled using a mean operation in the spatial dimension to obtain multi-layer difference features; -The classification head comprises: a two-layer fully connected network; wherein: The first layer of the fully connected network uses the ReLU activation function to enhance the nonlinear expression ability of features and distinguish between shifted and non-shifted patterns in the high-dimensional feature space; The second layer of the fully connected network performs the classification tasks of shift, non-shift and rejection through the Softmax classifier.

6. The camera displacement recognition method based on twin network according to claim 1 is characterized in that: The optimizing the multi-task model comprises: The multi-task model is optimized by using a cross entropy loss function combined with a contrast loss function; wherein: The cross entropy loss function is used to optimize the classification task of the multi-task model; The contrast loss function is used to calculate the Euclidean distance between the features of the previous and next frames and to perform weighted constraints on the shift, non-shift and rejection category labels, so that the multi-task model can distinguish the shift features in different scenarios.

7. The camera displacement recognition method based on twin network according to claim 6 is characterized in that: Also includes any one or more of the following: - the contrast loss function further includes: for the selected shift and rejection samples, the multi-task model treats the shift category and the rejection category as the shift category in the contrast loss calculation process, and calculates the feature distance in the embedding space through the contrast loss function to dynamically adjust the feature similarity; -The optimization of the multi-task model also includes: introducing a controllable weight parameter in the cross entropy loss and the contrast loss to balance the cross entropy loss and the contrast loss, so that the multi-task model can cope with the complexity of data distribution and the weight distribution problem between tasks while optimizing the shift recognition ability.

8. The camera displacement recognition method based on a twin network according to any one of claims 1 to 7, characterized in that: Also includes: The camera shift detection model is used to perform camera shift recognition on the image to be recognized.

9. A camera shift recognition system based on twin network, characterized in that: include: A data acquisition module, which is used to acquire a dataset of front and back frame images containing a variety of scenes; A model training module, which uses a twin network model to extract features from the image data of the previous and next frames to obtain multi-layer features of the previous and next frames; and uses a spatial attention mechanism to calculate feature differences of the multi-layer features of the previous and next frames to obtain multi-layer difference features; Performing differential feature fusion and classification on the multi-layer differential features, and training to obtain a multi-task model combining a twin network and a spatial attention mechanism; A model optimization module is used to optimize the multi-task model to obtain a camera shift detection model for performing camera shift recognition on the image to be recognized.

10. The camera shift recognition system based on twin network according to claim 9, characterized in that: Also includes: A shift recognition module uses the camera shift detection model to perform camera shift recognition on the image to be recognized.

11. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it can be used to perform the method described in any one of claims 1 to 8, or to run the system described in any one of claims 9 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method according to any one of claims 1 to 8, or to run the system according to any one of claims 9 to 10.

Images