Visual content identification method and device, equipment, medium and program product

Abstract

The invention discloses a visual content recognition method and device, equipment, a medium and a program product, and belongs to the field of artificial intelligence. The method comprises the steps that first visual content is acquired, the first visual content is used for representing visual content output when a first camera shoots, and the model of the first camera is a first model; inputting the first visual content into a feature extraction network to obtain a first feature vector; under the condition that the similarity between the first feature vector and a first center vector is smaller than a similarity threshold value, it is determined that the first visual content is the visual content used for attacking, and the first center vector is used for representing the center of the feature vector of at least one piece of visual content shot by the camera of the first model. According to the method, whether the first visual content is the visual content used for attacking or not is identified based on the similarity between the first feature vector of the first visual content and the first center vector, and noise does not need to be extracted as a camera fingerprint.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device, medium, and program product for recognizing visual content. Background Technology

[0002] Facial recognition technology refers to the technology that uses facial recognition to verify the authenticity of a user's identity information. For example, it can verify whether the user is a real person by analyzing their selfie.

[0003] In related technologies, in order to defend against attacks (such as synthesizing / editing / replacing user's selfie photos), facial recognition also uses camera fingerprints to verify the source of the selfie photo. Camera fingerprints refer to the unique noise generated by each camera when taking a photo due to the slight differences in its sensor and optical system.

[0004] However, extracting image noise that can be used as camera fingerprints using related technologies requires a large amount of computing and storage resources, making camera fingerprint extraction difficult and resource-intensive. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and program product for visual content recognition. This application extracts a first feature vector from the first visual content, and then identifies whether the first visual content is intended for attack based on the similarity between the first feature vector and a first center vector. This application eliminates the need to extract noise as a camera fingerprint, as noise extraction is a computationally and storage-intensive operation. The technical solution includes the following:

[0006] According to one aspect of this application, a method for recognizing visual content is provided, the method comprising the following steps.

[0007] Acquire first visual content, which is used to characterize the visual content output when captured by the first camera, and the model of the first camera is the first model;

[0008] The first visual content is input into the feature extraction network to obtain the first feature vector;

[0009] If the similarity between the first feature vector and the first center vector is less than a similarity threshold, the first visual content is determined to be the visual content used for the attack. The first center vector is used to characterize the center of the feature vector of at least one visual content captured by the camera of the first model.

[0010] According to another aspect of this application, a visual content recognition device is provided, the device comprising the following modules.

[0011] The acquisition module is used to acquire the first visual content, which represents the visual content output when the first camera is captured. The model of the first camera is the first model.

[0012] The feature extraction module is used to input the first visual content into the feature extraction network to obtain the first feature vector;

[0013] The determination module is used to determine the first visual content as visual content for attack when the similarity between the first feature vector and the first center vector is less than a similarity threshold. The first center vector is used to characterize the center of the feature vector of at least one visual content captured by the first type of camera.

[0014] According to one aspect of this application, a computer device is provided, comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the visual content recognition method described above.

[0015] According to another aspect of this application, a computer-readable storage medium is provided, which stores a computer program that is loaded and executed by a processor to implement the above-described method for recognizing visual content.

[0016] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned visual content recognition method.

[0017] The beneficial effects of the technical solutions provided in this application include at least the following:

[0018] In this application, feature extraction is performed based on the first visual content, and matching is then performed based on the extracted first feature vector, rather than matching through image noise. This application eliminates the need for a noise extraction step, which is extremely resource-intensive in terms of computation and storage. Therefore, the visual content recognition method provided in this application saves computational and storage resources. Furthermore, this application verifies the authenticity of the first visual content by detecting the camera model. When the first visual content is intended for attack, the method provided in this application can locate the device source of the first visual content, assisting law enforcement agencies in identifying malicious users.

[0019] In addition, the method of this application only requires one visual element for recognition, for example, only one image needs to be acquired, while related technologies require acquiring multiple images and averaging them to enhance the PRNU pattern. Therefore, the method of this application requires less data. Furthermore, related technologies rely on image noise as a camera fingerprint, but images need to undergo compression during transmission. While reducing the amount of image data, compression will process and change the details and noise patterns of the image. The PRNU pattern relies on these details and noise patterns for recognition, meaning that the PRNU pattern is sensitive to compression. However, this application does not use the PRNU pattern as a camera fingerprint, reducing the impact of compression on the final recognition result. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a visual content recognition system provided in one embodiment of this application.

[0022] Figure 2 This is a flowchart of a visual content recognition method provided in one embodiment of this application.

[0023] Figure 3 This is a flowchart of a visual content recognition method provided in another embodiment of this application.

[0024] Figure 4 This is a schematic diagram of the training objective of the first triplet loss provided in one embodiment of this application.

[0025] Figure 5 This is a schematic diagram of a training method based on the first triplet loss provided in one embodiment of this application.

[0026] Figure 6 This is a schematic diagram of a method for calculating the first ternary loss provided in one embodiment of this application.

[0027] Figure 7 This is a schematic diagram of a training method based on a first consistency loss provided in one embodiment of this application.

[0028] Figure 8 This is a schematic diagram of the training objective of the second triplet loss provided in one embodiment of this application.

[0029] Figure 9This is a schematic diagram of a training method based on second triplet loss provided in one embodiment of this application.

[0030] Figure 10 This is a schematic diagram of a method for calculating the second triplet loss provided in one embodiment of this application.

[0031] Figure 11 This is a schematic diagram of a training method based on a second consistency loss provided in one embodiment of this application.

[0032] Figure 12 This is a flowchart of an image recognition method provided in one embodiment of this application.

[0033] Figure 13 This is a structural block diagram of a visual content recognition device provided in one embodiment of this application.

[0034] Figure 14 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0037] Facial recognition verification refers to the technology used to verify the authenticity of user identity information, ensuring the legitimacy of user identities and preventing identity theft, forgery, and fraud. Typically, facial recognition technology includes techniques such as document OCR (Optical Character Recognition), liveness detection, and one-to-one face comparison to address the numerous user identity verification needs across various industries, such as real-name registration, password modification, and transaction withdrawals in sectors like finance, telecommunications, and ride-sharing.

[0038] Camera fingerprints refer to the unique noise patterns generated by each camera when capturing an image due to minute differences in its sensor and optical system. These noise patterns, similar to human fingerprints, can be used to identify the capturing device, hence the name "camera fingerprint." Camera fingerprints have important applications in digital forensics, copyright protection, and device authentication. Related technologies typically use the PRNU pattern as a camera fingerprint to verify image origin. The PRNU pattern arises because, due to minute defects and material inhomogeneities during the image sensor's manufacturing process, each pixel exhibits a slight difference in its response to light. This difference forms a unique noise pattern, called the PRNU pattern. The PRNU pattern is unique to each image sensor and can therefore be used as a camera fingerprint for image origin verification.

[0039] In related technologies, the camera fingerprint matching process is performed through the following steps.

[0040] 1. Image preprocessing: Perform preprocessing operations such as denoising and edge removal on the image output by the image sensor to reduce the influence of random noise and obtain a preprocessed image.

[0041] 2. Noise residual calculation: Subtract the estimated image content from the preprocessed image to obtain the noise residual.

[0042] 3. PRNU Pattern Extraction: Extract the PRNU pattern from the noise residual as a camera fingerprint. Some methods directly use the noise residual as the PRNU pattern, while others extract the feature representation of the PRNU pattern from the noise residual through deep learning.

[0043] 4. Averaging multiple images: The final PRNU mode is obtained by averaging multiple images to reduce the impact of random noise.

[0044] 5. Matching: Camera fingerprint matching of images is performed through similarity matching.

[0045] The camera fingerprint matching process of related technologies has the following drawbacks:

[0046] (1) Extracting PRNU patterns usually requires complex mathematical and statistical methods, such as wavelet transform and Fourier transform. Fourier transform is a commonly used method in frequency domain analysis, which requires Fast Fourier Transform (FFT) on the image, a computationally intensive operation. In practical applications, large amounts of image data are usually required to extract and match PRNU patterns, which requires a large amount of storage and computing resources.

[0047] (2) The matching process requires a large amount of data. To reduce the impact of random noise, the average value of multiple images is usually needed to enhance the final PRNU pattern.

[0048] (3) Sensitive to compression operations. Compression operations reduce the amount of image data, which will process and change the details and noise patterns of the image. The PRNU mode relies on these details and noise patterns for device recognition, and compression operations will affect the accuracy of the final recognition result.

[0049] Figure 1 This illustration shows a schematic diagram of a visual content recognition system provided in an exemplary embodiment of this application. The recognition system includes a first electronic device 10 and a backend server 11. The first electronic device 10 is equipped with a first camera, or the first electronic device 10 is externally connected (wirelessly or wired) to a first camera.

[0050] In response to receiving a shooting request, the first electronic device 10 will invoke the first camera to capture an image and output first visual content 110 through the first camera. The first visual content 110 can be an image or a video. The first visual content 110 may be real visual content captured by the first camera, or it may be visual content used by a malicious user for attack purposes, such as synthesized / edited / replaced visual content. The first electronic device 10 sends the first visual content 110 and the model number (first model number) of the first camera to the backend server 11, and the backend server 11 performs the visual content recognition process 100.

[0051] The backend server 11 stores center vectors corresponding to various camera models. Based on the first camera model, the backend server 11 obtains the first center vector 140 corresponding to that model. The first center vector 140 is used to characterize the center of the feature vector of at least one visual content captured by the camera of the first model. For example... Figure 1 As shown, the first center vector 140 is used to characterize the center of the feature vectors A, B and C corresponding to the three visual contents captured by the first type of camera.

[0052] In the visual content recognition process 100, the backend server 11 inputs the first visual content 110 into the feature extraction network 120 to obtain the first feature vector 130; calculates the similarity 150 between the first feature vector 130 and the first center vector 140, and then executes step 160: if the similarity 150 is less than the similarity threshold, the first visual content 110 is determined to be visual content used for attack.

[0053] If the similarity score is less than the similarity threshold, it means that the first feature vector 130 is not very similar to the first center vector 140. That is, the first visual content 110 is unlikely to be visual content captured by the first type of camera. Therefore, the first visual content 110 can be identified as visual content used for attack.

[0054] In this application, feature extraction is performed based on the first visual content, and matching is then performed based on the extracted first feature vector, rather than matching through image noise. This application eliminates the need for a noise extraction step, which is extremely resource-intensive in terms of computation and storage. Therefore, the visual content recognition method provided in this application saves computational and storage resources. Furthermore, this application verifies the authenticity of the first visual content by detecting the camera model. When the first visual content is intended for attack, the method provided in this application can locate the device source of the first visual content, assisting law enforcement agencies in identifying malicious users.

[0055] In addition, the method of this application only requires one visual element for recognition, for example, only one image needs to be acquired, while related technologies require acquiring multiple images and averaging them to enhance the PRNU pattern. Therefore, the method of this application requires less data. Furthermore, related technologies rely on image noise as a camera fingerprint, but images need to undergo compression during transmission. While reducing the amount of image data, compression will process and change the details and noise patterns of the image. The PRNU pattern relies on these details and noise patterns for recognition, meaning that the PRNU pattern is sensitive to compression. However, this application does not use the PRNU pattern as a camera fingerprint, reducing the impact of compression on the final recognition result.

[0056] In one embodiment, the first electronic device 10 may be at least one of a smartphone, smartwatch, in-vehicle terminal, wearable device, camera, smart TV, tablet computer, e-book reader, MP3 player, MP4 player, laptop computer, and desktop computer.

[0057] In one embodiment, the backend server 11 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0058] The visual content recognition method provided in this application can be applied to face verification scenarios.

[0059] In one embodiment, Figure 1The first electronic device 10 in the system has a first client installed. This first client requires facial recognition verification. For example, if the first client is a payment application, it needs to verify the user in a bank card withdrawal scenario to ensure that the withdrawal operation is performed by a real user. Or, if the first client is a financial application, it needs to verify the user in a loan scenario to ensure that the loan operation is performed by a real user. Still, if the first client is a government affairs and public service client, it needs to verify the user in certain transaction processing scenarios to ensure that the submitted transaction is performed by a real user, such as exam registration, driver's license application, business registration, etc.

[0060] Furthermore, besides the facial recognition scenario described above, the visual content recognition method provided in this application can also be applied to other scenarios, such as digital forensics, copyright protection, and device authentication. The visual content recognition method provided in this application can verify whether the first visual content is genuinely captured visual content; therefore, the method of this application can be applied in various scenarios requiring verification of the authenticity of visual content.

[0061] It should be noted that the information (including but not limited to user device information, user personal information, user photos, videos, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0062] Furthermore, when dealing with relevant information, the relevant information processor will adhere to the principles of legality, legitimacy, and necessity, clearly define the purpose, method, and scope of the relevant information processing, obtain the consent of the relevant information subject, and take necessary technical and organizational measures to ensure the security of the relevant information.

[0063] Figure 2 A flowchart illustrating a visual content recognition method provided in an exemplary embodiment of this application is shown, wherein the method is... Figure 1 The method is illustrated by example using the backend server 11 in the example, which includes:

[0064] Step 220: Obtain the first visual content. The first visual content is used to characterize the visual content output when the first camera is used to capture the image. The model of the first camera is the first model.

[0065] First-person view content (FPV) is used to represent the visual content output when captured by the first camera. In various scenarios, FPV may be the real visual content captured by the first camera; or it may be fake visual content that appears to be captured by the first camera but is actually used for attack purposes, such as visual content synthesized / edited / replaced by malicious users.

[0066] Optionally, the first visual content can be either an image or a video. For example, during facial recognition, the user can send a selfie or a selfie video to the backend server for identification. The identification method provided in this application supports both image and video recognition.

[0067] In one embodiment, the first visual content is the complete visual content output by the first camera when it takes a picture, such as the entire photo or video output by the first camera. In this case, the present application can verify the authenticity based on more visual content information. When a certain image block is abnormal, it can be determined that there is an attack risk. Verification based on more visual content information ensures the accuracy of the final recognition result.

[0068] In another embodiment, the first visual content is a portion of the visual content output by the first camera during capture. For example, it could be one or more image blocks from the entire photograph output by the first camera, or one or more image blocks at fixed positions obtained by upsampling the entire video output by the first camera, such as the image block located in the upper left corner of image frames 1 to N in the video. In this case, the present application uses less visual content information to verify authenticity, reducing data processing and storage, thereby improving the efficiency of visual content recognition.

[0069] The first camera can be a standalone camera, such as a portable mobile camera. These cameras are small and easy to carry, offering convenience for certain shooting scenarios. Alternatively, the first camera can be a camera integrated into an electronic device, such as a smartphone, camera, or tablet. Typically, manufacturers specify the model number of the camera when it leaves the factory, and each manufacturer has its own naming conventions for these models.

[0070] Step 240: Input the first visual content into the feature extraction network to obtain the first feature vector;

[0071] A feature extraction network is a neural network used to extract features from visual content. More specifically, a feature extraction network is a neural network used to extract image or video features. Optionally, the feature extraction network is a deep feature extraction network, which has multiple hidden layers and can extract information hidden deep within the visual content, thus obtaining higher-quality features of the visual content.

[0072] Optional deep feature extraction networks include RestNet, DenseNet, etc.

[0073] In one embodiment, a global average pooling (GAP) layer is placed before the last fully connected layer of the deep feature extraction network. In the deep feature extraction network, a feature map is extracted through multiple convolutional layers. This feature map is represented as H*W*C (H represents the height of the image, W represents the width of the image, and C represents the number of channels in the feature map). Then, the global average pooling layer pools this feature map into an intermediate vector. The global average pooling layer averages the values ​​in each channel to obtain an intermediate vector represented as 1*C. Finally, this intermediate vector is input into a fully connected layer, which flattens the intermediate vector into a one-dimensional vector.

[0074] In this embodiment, a global average pooling layer is set before the last fully connected layer, enabling the feature extraction network to support input visual content of different sizes. For visual content of different sizes, it will eventually be transformed into a 1*C feature map through the global average pooling layer, and then the 1*C feature map will be input into the fully connected layer, avoiding the problem of mismatch between the parameter dimension of the fully connected layer and the dimension of the input feature map.

[0075] Step 260: If the similarity between the first feature vector and the first center vector is less than a similarity threshold, the first visual content is determined to be the visual content for attack. The first center vector is used to characterize the center of the feature vector of at least one visual content captured by the first type of camera.

[0076] For the extracted first feature vector, the similarity between the first feature vector and the first center vector corresponding to the first model is used to determine whether the first visual content is intended for attack. If the similarity between the first feature vector and the first center vector is less than a similarity threshold, the first visual content is determined to be intended for attack; conversely, if the similarity between the first feature vector and the first center vector is not less than the similarity threshold, the first visual content is determined to be genuine visual content. Optionally, cosine similarity can be used to calculate the similarity between the first feature vector and the first center vector.

[0077] The first center vector is the center vector corresponding to the first model. In one embodiment, multiple center vectors corresponding to multiple camera models are pre-stored. After obtaining the first model, the center vector corresponding to the first model is matched from the multiple center vectors, which is the first center vector.

[0078] The first center vector corresponds to a specific camera model; different camera models correspond to different center vectors. The first center vector is used to characterize the center of the feature vector of at least one visual content captured by the camera of model number one. In one embodiment, the camera of model number one includes multiple camera devices, and the first center vector is used to characterize the center of the feature vector of the visual content captured by the multiple camera devices, each camera device having a unique device number.

[0079] The first center vector is used to characterize the center of the feature vector of at least one visual content captured by the camera of the first type. When the at least one visual content is multiple visual content, the first center vector is used to characterize the average value of the multiple feature vectors corresponding to the multiple visual content. When the at least one visual content is a single visual content, the first center vector is used to characterize the feature vector itself corresponding to that visual content.

[0080] In one embodiment, different camera models correspond to the same similarity threshold. Optionally, the similarity threshold is determined by considering the similarity dispersion of visual content captured by multiple camera models. For example, for camera model A, the similarity between any two visual contents captured by it is above 90%, while for camera model B, the similarity between any two visual contents captured by it is above 86%, and for camera model C, the similarity between any two visual contents captured by it is above 88%. Therefore, considering all factors, the similarity threshold is set to 88%.

[0081] In another embodiment, different camera models correspond to different similarity thresholds. For example, for some camera models on the market with unstable shooting effects, a special similarity threshold is set for these camera models. The feature vectors of the visual content captured by these camera models have large deviations, and setting a special similarity threshold can more accurately determine whether the visual content output by these cameras has been attacked.

[0082] based on Figure 2 In the alternative embodiments shown, Figure 3 Two possible implementations of step 260 are shown. The first possible implementation includes steps 261 and 262, and the second possible implementation includes steps 263 to 266.

[0083] First possible implementation:

[0084] Step 261: If the first model belongs to the camera models in the training set of the feature extraction network, obtain the first center vector obtained when training the feature extraction network;

[0085] If the first model belongs to the camera models in the training set of the feature extraction network, the first center vector corresponding to the first model will be obtained from the pre-stored data. In one embodiment, the first center vector is a portion of the weight parameters of the last fully connected layer in the feature extraction network that corresponds to the first model, and the weight parameters include at least two center vectors corresponding to at least two camera models.

[0086] In the trained feature extraction network, the weight parameters W of the last fully connected layer correspond exactly to the center vectors of multiple camera models. The weight parameters of the last fully connected layer are represented as W∈R d*n Here, n represents n camera models, and d represents a center vector corresponding to one camera model. Through the trained feature extraction network, the center vectors corresponding to each of the multiple camera models can be obtained. This application obtains the center vectors from the weight parameters of the fully connected layer, ensuring accurate utilization of the fully connected layer's weight parameters and improving their utilization rate. Furthermore, this application eliminates the need for additional computational steps to calculate the center vectors corresponding to each camera model; it only requires obtaining the weight parameters of the fully connected layer.

[0087] After the feature extraction network completes training, the weight parameters of the last fully connected layer of the feature extraction network are obtained, thereby obtaining the center vectors corresponding to each of the multiple camera models. Based on the obtained first model, the first center vector corresponding to the first model is determined.

[0088] Step 262: If the similarity between the first feature vector and the first center vector is less than the similarity threshold, the first visual content is determined to be the visual content used for the attack.

[0089] If the similarity between the first feature vector and the first center vector is less than a similarity threshold, the first visual content is determined to be the visual content used for attack. Since the first center vector is exactly the center vector that can be directly obtained during model training, the center vector obtained by the model training is more accurate when the training set contains a large number of training samples. Therefore, the accuracy of determining whether the first visual content is the attack visual content is high in this case. Furthermore, it is known that to obtain even higher accuracy, visual content captured by more types of cameras can be used for training to increase the coverage of the training set. In this case, the first center vector is learned from the training set by the feature extraction network, without the need for additional steps to calculate the first center vector. Therefore, the calculation of the first center vector is not only more accurate but also does not require additional acquisition methods.

[0090] The second possible implementation method:

[0091] Step 263: If the first model is not a camera model in the training set of the feature extraction network, acquire the first number of visual contents captured by the first camera.

[0092] If the first model is not a camera model in the training set of the feature extraction network, the first center vector cannot be obtained directly and will be obtained through an additional calculation process.

[0093] If the first model is not a camera model in the training set of the feature extraction network, the backend server notifies the first electronic device that it needs to collect a first number of visual contents. The first electronic device activates the first camera to collect the first number of visual contents, such as acquiring three images, five images, three videos, five videos, etc. The first electronic device sends the collected first number of visual contents to the backend server.

[0094] Step 264: Input the first number of visual contents into the feature extraction network to obtain the first number of visual content features, and the first number of visual content features correspond one-to-one with the first number of visual contents;

[0095] The feature extraction network is used to extract features from visual content. A first set of visual content is input into the feature extraction network, which then outputs the visual content features corresponding to each of the first set of visual content.

[0096] Step 265: Based on the average value of the first number of visual content features, obtain the first center vector;

[0097] A first center vector is obtained based on the average value of a first number of visual content features. Optionally, a normalization operation is first performed on the first number of visual content features, and then the average value of the normalized first number of visual content features is calculated to obtain the first center vector.

[0098] Step 266: Based on the fact that the similarity between the first feature vector and the first center vector is less than the similarity threshold, the first visual content is determined to be the visual content used for the attack.

[0099] If the similarity between the first feature vector and the first center vector is less than a similarity threshold, the first visual content is determined to be visual content used for attack. When faced with an unknown camera model, this application calculates the first center vector by additionally acquiring a first number of visual contents, so that for any camera model, this application can determine whether its output visual content is visual content used for attack.

[0100] It should be noted that determining whether the first visual content is attack content through the above method relies on the opening set capability of the feature extraction network. In this application, the opening set capability refers to the ability of the feature extraction network to extract features of visual content captured by unknown camera models when processing unseen camera models. The feature extraction network trained using the method provided in this application has superior opening set capability, thus supporting the second possible implementation method. The training method will be described below.

[0101] The training method for the feature extraction network provided in this application will be introduced next. In this application, a pre-training plus fine-tuning approach is used to train the feature extraction network.

[0102] Next, we will introduce the fine-tuning stage of the model.

[0103] In this application, a first triplet loss and a first consistency loss are used to fine-tune the feature extraction network. The first triplet loss is used to bring visual content captured by the same camera model closer together and to widen the distance between visual content captured by different camera models. The first consistency loss is used to make the feature vectors corresponding to different cropped content on the same visual content consistent.

[0104] Regarding the loss of the first triplet

[0105] like Figure 4 As shown, Figure 4 The diagram illustrates the principle of fine-tuning the feature extraction network using the first triplet loss. When fine-tuning the feature extraction network using the first triplet loss, the training objective is to reduce the distance between the sample visual content captured by a camera of a target model and the visual content category corresponding to the target model, while simultaneously increasing the distance between the sample visual content and the visual content categories corresponding to other models. This approach can simultaneously improve both intra-class clustering and inter-class differentiation.

[0106] Specifically, Figure 5 This illustration shows a schematic diagram of a method for fine-tuning a feature extraction network according to an illustrative embodiment of this application. Second visual content 510 is acquired, which is visual content captured by a second camera, the second camera being a second model.

[0107] The second visual content 510 is input into the feature extraction network 520 to obtain the second feature vector 530;

[0108] Based on the similarity between the second feature vector 530, the second center vector 540, and the third center vector 550, a first triplet loss 560 is obtained. The second center vector 540 is used to characterize the center of the feature vector of at least one visual content captured by the second type of camera, and the third center vector 550 is used to characterize the center of the feature vector of at least one visual content captured by cameras of other types besides the second type. Based on the first triplet loss 560, the feature extraction network 520 is fine-tuned.

[0109] The first triplet loss aims to increase the similarity between the second feature vector and the second center vector, and decrease the similarity between the second feature vector and the third center vector. For cosine similarity, the first triplet loss is used to decrease the angle between the second feature vector and the second center vector, and to increase the angle between the second feature vector and the third center vector. A larger angle between two vectors indicates less similarity, while a smaller angle indicates greater similarity. Illustratively, the first triplet loss can be ArcFace loss, CosFace loss, SphereFace loss, Softmax loss, etc.

[0110] The ArcFace loss, which is the first triplet loss, will be used as an example for explanation.

[0111] like Figure 6 As shown, based on the similarity between the second feature vector 610 and the second center vector 620, a first angle 630 is obtained, which is used to characterize the angle between the second feature vector 610 and the second center vector 620; and based on the similarity between the second feature vector 610 and the third center vector 640, a second angle 650 is obtained, which is used to characterize the angle between the second feature vector 610 and the third center vector 640; based on the sum of the first angle 630 and the preset angle 660, a first angle sum 670 is obtained; based on the first angle sum 670 and the second angle 650, a first triplet loss 680 is obtained.

[0112] Optionally, for the first angle sum 670 and the second angle 650, cosine similarity is calculated respectively. Based on the calculated cosine similarity, a first triplet loss 680 is obtained. The first triplet loss 680 is used to increase the cosine similarity corresponding to the first angle sum 670 and to decrease the cosine similarity corresponding to the second angle 650.

[0113] For cosine similarity, the larger the angle between two vectors, the less similar they are; conversely, the smaller the angle, the more similar they are. Cosine similarity is denoted as cosθ. In ArcFace loss, an additional preset angle (which can be called additive margin) is added to the cosine similarity between the second feature vector and the second center vector, denoted as cos(θ+m).

[0114] By increasing the preset angle m, the value of cos(θ+m) will naturally be less than the value of cosθ. The loss value calculated based on the value of cos(θ+m) will naturally be greater than the loss value calculated based on cosθ. The training of the model is carried out in the direction of reducing the loss. That is, by increasing the preset angle m, the difficulty of reducing the loss is increased. As a result, when the model converges, the model can better achieve the goal, and the trained model will perform better.

[0115] Schematic, the loss of the first triplet is expressed as follows: (1)

[0116]

[0117] Where N is the number of camera models in an input batch of the feature extraction network, s is the scaling factor (usually set to a fixed value, such as 64), and m is the preset angle mentioned above (usually set to a fixed value, such as 0.5). It is the first angle mentioned above; θ j This refers to the second angle mentioned above, where j represents other models.

[0118] For the included angle and θ j It can be obtained by using the inverse cosine of the following cosine value.

[0119] in,

[0120]

[0121]

[0122] Where T represents the matrix transpose operation, x i This is the second eigenvector mentioned above. W is the second center vector mentioned above. j Let be the third center vector mentioned above. Using formulas (2) and (3), the inverse cosine can be calculated to obtain the first angle. Second angle θ j .

[0123] Regarding the first consistency loss

[0124] The first consistency loss aims to increase the consistency of features extracted from different cropped content within the same sample visual content. Combined with the first triplet loss introduced above, it can also reduce the consistency of features of different cropped content corresponding to different sample visual content.

[0125] like Figure 7 As shown, Figure 7 A schematic diagram of a method for fine-tuning a feature extraction network provided by an exemplary embodiment is shown. Figure 7 The second visual content 710 and the third visual content 720 both originate from the same sample visual content, referred to as the first sample visual content. The first sample visual content is complete visual content, such as a complete image or a complete video. The second visual content 710 and the third visual content 720 are cropped content from the first sample visual content. The second visual content 710 is input into the feature extraction network 730 to obtain the second feature vector 740. The third visual content 720 is input into the feature extraction network 730 to obtain the third feature vector 750. Based on the feature distance between the second feature vector 740 and the third feature vector 750, a first consistency loss 760 is obtained. The feature extraction network 730 is then fine-tuned based on the first consistency loss 760.

[0126] Figure 7 The method for calculating the first consistency loss for a single sample visual content is illustrated. Optionally, the first consistency loss is calculated based on multiple sample visual contents in an input batch of the model. Optionally, the feature distance between the second and third feature vectors is calculated as the feature distance corresponding to the first sample visual content, and the first consistency loss is obtained by using at least two feature distances corresponding to at least two sample visual contents.

[0127] Illustratively, the formula for calculating the first consistency loss is expressed as follows: (4)

[0128]

[0129] Where x1 and x2 represent different cropped contents of the same sample visual content, i represents the i-th sample visual content, and D represents the number of sample visual contents included in a batch, i.e., the number of complete images included. This represents the feature distance between different cropped contents of the same sample visual content.

[0130] Training the feature extraction network using the first consistency loss can effectively accelerate the model's training speed. When training with the first triplet loss, the feature extraction network receives multiple different visual samples as input. These samples vary significantly, resulting in slow learning speed and high training difficulty. However, when training with the first consistency loss, the network receives different cropped versions of a single visual sample as input. These cropped versions share similarities, allowing the network to learn much faster from the same visual sample. In the early stages of model training, training solely with the first triplet loss is challenging, while incorporating the first consistency loss reduces this early training difficulty.

[0131] Furthermore, the first consistency loss increases the diversity of training, enabling the trained feature extraction network to classify different cropped content from the same visual content, thus providing a more comprehensive capability. Moreover, training with the first consistency loss constrains the feature extraction network, ensuring that the feature vectors corresponding to different cropped content from the same sample visual content are consistent. This allows the network to learn the similarity within the same sample visual content, enabling it to accurately classify a cropped content as originating from sample visual content A or sample visual content B. Consequently, the network is better able to determine whether the cropped content was captured by camera model A (including sample visual content A) or camera model B (including sample visual content B). In other words, training with the first consistency loss allows the feature extraction network to better achieve intra-class clustering within the same camera model and inter-class differentiation between different camera models.

[0132] Furthermore, in this application, the model training is constrained by the first consistency loss obtained from the overall visual content of multiple samples, that is, the model parameter adjustment is constrained so that the model parameter adjustment fully considers the overall visual content of multiple samples, and the parameter adjustment of the model will be more accurate and more balanced.

[0133] Points that also need to be explained

[0134] 1. The above text describes how the feature extraction network is fine-tuned using the first triplet loss and the first consistency loss. In one embodiment, the first triplet loss and the first consistency loss are weighted and fused to obtain a fusion loss, which is then used to fine-tune the feature extraction network.

[0135] 2. In one embodiment, when fine-tuning the feature extraction network, visual content for training is input into the feature extraction network in batches. The size of the visual content in different batches is different, while the size of the visual content in the same batch is the same. Optionally, the size of the visual content in each batch is random. For example, in a batch, the second and third visual content from the same sample visual content have the same size, which is randomly obtained from [w / 2, w], where w is the size of the source sample visual content.

[0136] The pre-training phase of the model will be introduced next.

[0137] In this application, the pre-training phase is similar to the fine-tuning phase. During the fine-tuning phase, the training objective is to correctly classify the sample visual content to the visual content corresponding to the target camera model, while simultaneously distinguishing the sample visual content from the visual content corresponding to other camera models.

[0138] In the pre-training phase, the training objective is to correctly classify sample visual content into the visual content corresponding to the target camera device, while also distinguishing the sample visual content from the visual content corresponding to other camera devices. Within the same camera model, there are a relatively large number of camera devices, each with a unique device number. That is, in the pre-training phase, the feature extraction network will utilize more categories (one camera device per category) to learn a broader range of capabilities, while the fine-tuning phase will utilize fewer categories (one camera model per category) to learn the ability to perform a specific task.

[0139] In this application, a feature extraction network is pre-trained using a second triplet loss and a second consistency loss. The second triplet loss is used to bring visual content captured by the same camera device closer together and to widen the distance between visual content captured by different camera devices. The second consistency loss is used to make the feature vectors corresponding to different cropped content on the same visual content consistent.

[0140] Regarding the loss of the second triplet

[0141] like Figure 8 As shown, Figure 8 The diagram illustrates the principle of pre-training the feature extraction network using the second triplet loss. When pre-training the feature extraction network using the second triplet loss, the training objective is to reduce the distance between the sample visual content captured by a target device and the corresponding visual content category of the target device, while simultaneously increasing the distance between the sample visual content and the corresponding visual content categories of other devices. This approach can simultaneously improve both intra-class clustering and inter-class differentiation.

[0142] Specifically, Figure 9This illustration shows a schematic diagram of a method for pre-trained feature extraction networks provided in an illustrative embodiment of this application. Fourth visual content 910 is acquired, which is visual content captured by a third camera, the device number of which is the first device number.

[0143] The fourth visual content 910 is input into the feature extraction network 920 to obtain the fourth feature vector 930;

[0144] Based on the similarity between the fourth feature vector 930, the fourth center vector 940, and the fifth center vector 950, a second triplet loss 960 is obtained. The fourth center vector 940 is used to characterize the center of the feature vector of at least one visual content captured by the camera with the first device number, and the fifth center vector 950 is used to characterize the center of the feature vector of at least one visual content captured by cameras with other device numbers besides the first device number. Based on the second triplet loss 960, a pre-trained feature extraction network 920 is trained.

[0145] The second triplet loss aims to increase the similarity between the fourth feature vector and the fourth center vector, and decrease the similarity between the fourth feature vector and the fifth center vector. For cosine similarity, the second triplet loss aims to decrease the angle between the fourth feature vector and the fourth center vector, and increase the angle between the fourth feature vector and the fifth center vector. A larger angle between two vectors indicates less similarity, while a smaller angle indicates greater similarity. Illustratively, the second triplet loss can be ArcFace loss, CosFace loss, SphereFace loss, Softmax loss, etc.

[0146] The ArcFace loss, which is the second triplet loss, will be used as an example for explanation.

[0147] like Figure 10 As shown, based on the similarity between the fourth feature vector 1010 and the fourth center vector 1020, a third angle 1030 is obtained, which is used to characterize the angle between the fourth feature vector 1010 and the fourth center vector 1020; and based on the similarity between the fourth feature vector 1010 and the fifth center vector 1040, a fourth angle 1050 is obtained, which is used to characterize the angle between the fourth feature vector 1010 and the fifth center vector 1040; based on the sum of the third angle 1030 and the preset angle 1060, a second angle sum 1070 is obtained; based on the second angle sum 1070 and the fourth angle 1050, a second triplet loss 1080 is obtained.

[0148] Optionally, for the second angle sum of 1070 and the fourth angle of 1050, cosine similarity is calculated respectively. Based on the calculated cosine similarity, a second triplet loss of 1080 is obtained. The second triplet loss of 1080 is used to increase the cosine similarity corresponding to the second angle sum of 1070 and to decrease the cosine similarity corresponding to the fourth angle of 1050.

[0149] For cosine similarity, the larger the angle between two vectors, the less similar they are; conversely, the smaller the angle, the more similar they are. Cosine similarity is denoted as cosθ. In ArcFace loss, for the cosine similarity between the fourth feature vector and the fourth center vector, an additional preset angle (which can be called the additive margin) is added, denoted as cos(θ+m).

[0150] By increasing the preset angle m, the value of cos(θ+m) will naturally be less than the value of cosθ. The loss value calculated based on the value of cos(θ+m) will naturally be greater than the loss value calculated based on cosθ. The training of the model is carried out in the direction of reducing the loss. That is, by increasing the preset angle m, the difficulty of reducing the loss is increased. As a result, when the model converges, the model can better achieve the goal, and the trained model will perform better.

[0151] Schematic, the loss of the second triplet is expressed as follows: (5)

[0152]

[0153] Where N is the number of camera devices in an input batch of the feature extraction network, s is the scaling factor (usually set to a fixed value, such as 64), and m is the preset angle mentioned above (usually set to a fixed value, such as 0.5). It is the third angle mentioned above, θ j This is the fourth angle mentioned above, where j is the number of other devices.

[0154] For the included angle and θ j It can be obtained by using the inverse cosine of the following cosine value.

[0155] in,

[0156]

[0157]

[0158] Where T represents the matrix transpose operation, x i This is the fourth eigenvector mentioned above. W is the fourth center vector mentioned above. jLet be the fifth central vector mentioned above. Using formulas (6) and (7), the inverse cosine can be calculated to obtain the third angle. and the fourth angle θ j .

[0159] It should also be noted that within the same camera model, there are a relatively large number of camera devices, each with a unique device number. This means that in the pre-training phase of this application, the feature extraction network will utilize more categories (one camera device per category) to learn a broader range of capabilities, while the fine-tuning phase will utilize fewer categories (one camera model per category) to learn the ability to perform a specific task. Utilizing more categories in the pre-training phase improves the model's generalization and open-set capabilities.

[0160] Furthermore, this application will use the camera model to perform the model's reasoning process. When the model is reasoning, it will determine whether the first visual content obtained falls into the visual content category corresponding to the corresponding camera model. Compared to using the camera's device number for reasoning, the camera model is information that can be obtained under some data privacy requirements, and it is easier to obtain the camera model than to obtain the camera's device number.

[0161] Furthermore, if the first feature vector extracted by the feature extraction network is the feature vector corresponding to the camera model, the feature extraction accuracy corresponding to the first feature vector is relatively coarse, and the feature extraction operation of the feature extraction network is relatively easy to implement; if the first feature vector is the feature vector corresponding to the camera device number, and one camera model includes multiple device numbers, the feature extraction accuracy corresponding to the first feature vector is relatively fine, and the feature extraction operation of the feature extraction network is more difficult to implement.

[0162] Furthermore, assuming the training set of the feature extraction network includes the registered visual content sent by the first camera, when the first camera sends the first visual content to be identified to the backend and extracts the first feature vector, the first center vector corresponding to the camera model of the first camera will be used to match the first feature vector, instead of using the feature vector corresponding to the device number of the first camera. This avoids the situation where the registered visual content sent by the first camera during registration has been attacked, resulting in incorrect matching. A camera model includes multiple device numbers, and the first center vector corresponding to the camera model of the first camera is used for matching. Even if the registered visual content sent by the first camera has been attacked, it will not have a significant impact on the first center vector.

[0163] Regarding the second consistency loss

[0164] The second consistency loss aims to increase the consistency of features extracted from different cropped content within the same sample visual content. Combined with the second triplet loss introduced above, it can also reduce the consistency of features of different cropped content corresponding to different sample visual content.

[0165] like Figure 11 As shown, Figure 11 A schematic diagram of a method for pre-trained feature extraction networks provided by an exemplary embodiment is shown. Figure 11 The fourth visual content 1110 and the fifth visual content 1120 come from the same sample visual content, referred to as the second sample visual content. The second sample visual content is complete visual content, such as a complete image or a complete video. The fourth visual content 1110 and the fifth visual content 1120 are cropped content from the second sample visual content. The fourth visual content 1110 is input into the feature extraction network 1130 to obtain the fourth feature vector 1140, and the fifth visual content 1120 is input into the feature extraction network 1130 to obtain the fifth feature vector 1150. Based on the feature distance between the fourth feature vector 1140 and the fifth feature vector 1150, the second consistency loss 1160 is obtained. The feature extraction network 1130 is pre-trained based on the second consistency loss 1160.

[0166] Figure 11 The method for calculating the second consistency loss for a single sample visual content is illustrated. Optionally, the second consistency loss is calculated based on multiple sample visual contents in an input batch of the model. Optionally, the feature distance between the fourth and fifth feature vectors is calculated as the feature distance corresponding to the second sample visual content, and the second consistency loss is obtained by using at least two feature distances corresponding to at least two sample visual contents.

[0167] As an illustration, the formula for calculating the second consistency loss is as follows (8):

[0168]

[0169] Where x1 and x2 represent different cropped contents of the same sample visual content, i represents the i-th sample visual content, and D represents the number of sample visual contents in a batch. This represents the feature distance between different cropped contents of the same sample visual content.

[0170] Training the feature extraction network using the second consistency loss can effectively accelerate the model's training speed. When training with the second triplet loss, the feature extraction network receives multiple different visual samples as input. These samples vary significantly, resulting in slow learning speed and high training difficulty. However, when training with the second consistency loss, the network receives different cropped versions of a single visual sample as input. These cropped versions share similarities, allowing the network to learn much faster from the same visual sample. In the early stages of model training, training solely with the second triplet loss is challenging; incorporating the second consistency loss reduces this early training difficulty.

[0171] Furthermore, the second consistency loss increases the diversity of training, resulting in a more comprehensive feature extraction network. Training with the second consistency loss also constrains the feature extraction network, ensuring that the feature vectors corresponding to different cropped content within the same sample visual content are consistent. This allows the feature extraction network to learn the similarity within the same sample visual content, enabling it to accurately classify a cropped content as originating from sample visual content A or sample visual content B. Consequently, the feature extraction network is better able to discern whether the cropped content was captured by camera device A (which includes sample visual content A) or camera device B (which includes sample visual content B). In other words, training with the second consistency loss allows the feature extraction network to better achieve intra-class clustering and inter-class differentiation within the same camera device.

[0172] Furthermore, in this application, the model training is constrained by the second consistency loss obtained from the overall visual content of multiple samples, that is, the model parameter adjustment is constrained so that the model parameter adjustment fully considers the overall visual content of multiple samples, and the parameter adjustment of the model will be more accurate and more balanced.

[0173] Points that also need to be explained

[0174] 1. The above describes the pre-training of the feature extraction network using the second triplet loss and the second consistency loss. In one embodiment, the second triplet loss and the second consistency loss are weighted and fused to obtain a fused loss, which is then used to pre-train the feature extraction network.

[0175] 2. In one embodiment, during the pre-training of the feature extraction network, visual content for training is input into the feature extraction network in batches. The size of the visual content in different batches is different, while the size of the visual content in the same batch is the same. Optionally, the size of the visual content in each batch is random. For example, in a batch, the second and third visual contents from the same sample visual content have the same size, which is randomly obtained from [w / 2, w], where w is the size of the source sample visual content.

[0176] Application Scenario Examples

[0177] Figure 12 A flowchart of an image recognition method provided in an exemplary embodiment of this application is shown, in which the method is... Figure 1 The method is illustrated by example using the backend server 11 in the example, which includes:

[0178] Step 1220: In response to the face verification request, obtain the first image. The first image is used to represent the image output when the first camera is captured. The model of the first camera is the first model.

[0179] Optionally, the first image is the image output by the first electronic device when it receives a face verification request and calls the first camera to capture the image. Optionally, the first image contains a face. The first visual content may be a real image captured by the first camera, or it may be a fake image that appears to be captured by the first camera but is actually used for attack purposes, such as an image obtained by a malicious user by synthesizing / editing / replacing a real image.

[0180] In one embodiment, the first image is a complete image output when the first camera captures the image. In this case, the present application can verify authenticity based on more image information, and verification based on more image information ensures the accuracy of the final recognition result.

[0181] In another embodiment, the first image is a partial image block output by the first camera when it takes a picture, such as one or more image blocks from the entire photograph output by the first camera. In this case, the present application uses less image information to verify authenticity, which can reduce data processing and storage, thereby improving the efficiency of image recognition.

[0182] Step 1240: Input the first image into the feature extraction network to obtain the first feature vector;

[0183] The first image is input into the feature extraction network to obtain the feature vector corresponding to the first image, i.e., the first feature vector. Optionally, the feature extraction network is a deep feature extraction network, such as RESTNet or DenseNet.

[0184] Step 1260: If the similarity between the first feature vector and the first center vector is less than a similarity threshold, the first image is determined to be the image used for attack. The first center vector is used to characterize the center of the feature vector of at least one image captured by the first type of camera.

[0185] For the extracted first feature vector, the similarity between the first feature vector and the first center vector corresponding to the first model is used to determine whether the first image is used for attack. If the similarity between the first feature vector and the first center vector is less than a similarity threshold, the first image is determined to be used for attack; otherwise, if the similarity between the first feature vector and the first center vector is not less than the similarity threshold, the first image is determined to be a real image. Optionally, cosine similarity can be used to calculate the similarity between the first feature vector and the first center vector.

[0186] In one embodiment, if the first model belongs to the camera models in the training set of the feature extraction network, a first center vector obtained during the training of the feature extraction network is acquired; if the similarity between the first feature vector and the first center vector is less than a similarity threshold, the first image is determined to be the image used for the attack. Optionally, the first center vector is a portion of the weight parameters of the last fully connected layer in the feature extraction network that corresponds to the first model, and the weight parameters include center vectors corresponding to at least two camera models respectively.

[0187] In one embodiment, if the first model is not a camera model in the training set of the feature extraction network, a first number of images captured by the first camera are acquired; the first number of images are input into the feature extraction network to obtain a first number of image features, and the first number of image features correspond one-to-one with the first number of images; a first center vector is obtained based on the average value of the first number of image features; and the first image is determined to be an image used for attack based on the similarity between the first feature vector and the first center vector being less than a similarity threshold.

[0188] In one embodiment, the feature extraction network is trained through the following steps:

[0189] Acquire a second image, which is an image captured by a second camera, the model of which is a second model;

[0190] The second image is input into the feature extraction network to obtain the second feature vector;

[0191] Based on the similarity between the second feature vector, the second center vector, and the third center vector, a first triplet loss is obtained. The second center vector is used to characterize the center of the feature vector of at least one image captured by the second type of camera, and the third center vector is used to characterize the center of the feature vector of at least one image captured by cameras of other types besides the second type.

[0192] The feature extraction network is trained based on the first triplet loss.

[0193] In one embodiment, a first angle is obtained based on the similarity between the second feature vector and the second center vector; and a second angle is obtained based on the similarity between the second feature vector and the third center vector; the first angle is used to characterize the angle between the second feature vector and the second center vector, and the second angle is used to characterize the angle between the second feature vector and the third center vector;

[0194] The sum of the first angle and the preset angle is used to obtain the sum of the first angles.

[0195] Based on the first angle and the second angle, the first triplet loss is obtained.

[0196] In one embodiment, the second image is a cropped image patch from the first sample image; the method further includes:

[0197] Obtain the third image, which is a cropped image patch on the first sample image that is different from the second image;

[0198] The third image is input into the feature extraction network to obtain the third feature vector;

[0199] The first consistency loss is obtained based on the feature distance between the second and third feature vectors; the feature extraction network is trained based on the first consistency loss.

[0200] In one embodiment, the first sample image is one of at least two sample images; the feature distance between the second feature vector and the third feature vector is calculated;

[0201] The first consistency loss is obtained based on the distance between at least two features corresponding to at least two sample images.

[0202] In one embodiment, a fourth image is acquired, which is an image captured by a third camera, and the device number of the third camera is the same as the first device number.

[0203] The fourth image is input into the feature extraction network to obtain the fourth feature vector;

[0204] Based on the similarity between the fourth feature vector, the fourth center vector, and the fifth center vector, a second triplet loss is obtained. The fourth center vector is used to characterize the center of the feature vector of at least one image captured by the camera with the first device number, and the fifth center vector is used to characterize the center of the feature vector of at least one image captured by the camera with other device numbers besides the first device number.

[0205] The feature extraction network is trained based on the second triplet loss.

[0206] In one embodiment, a third angle is obtained based on the similarity between the fourth feature vector and the fourth center vector; and a fourth angle is obtained based on the similarity between the fourth feature vector and the fifth center vector; the third angle is used to characterize the angle between the fourth feature vector and the fourth center vector, and the fourth angle is used to characterize the angle between the fourth feature vector and the fifth center vector.

[0207] The sum of the third angle and the preset angle is used to obtain the sum of the second angle.

[0208] Based on the second angle sum and the fourth angle, the second triplet loss is obtained.

[0209] In one embodiment, the fourth image is a cropped image patch from the second sample image; the method further includes:

[0210] Obtain the fifth image, which is a cropped image patch on the second sample image that is different from the fourth image;

[0211] The fifth image is input into the feature extraction network to obtain the fifth feature vector;

[0212] The second consistency loss is obtained based on the feature distance between the fourth and fifth feature vectors; the feature extraction network is trained based on the second consistency loss.

[0213] In one embodiment, the second sample image is one of at least two sample images; the feature distance between the fourth feature vector and the fifth feature vector is calculated;

[0214] The second consistency loss is obtained based on the distance between at least two features corresponding to at least two sample images.

[0215] In one embodiment, a global average pooling layer is placed before the last fully connected layer of the feature extraction network.

[0216] It should be noted that for more detailed information on the training methods of feature extraction networks, please refer to the relevant introduction above. Due to space limitations, it will not be elaborated further here.

[0217] In summary, this application extracts features from the first image and then performs matching based on the extracted first feature vector, rather than matching through image noise. This eliminates the need for a noise extraction step, which is highly computationally and storage-intensive. Therefore, the image recognition method provided in this application saves computational and storage resources. Furthermore, this application verifies the authenticity of the first image by detecting the camera model. When the first image is used for attack purposes, the method provided in this application can locate the device source of the first image, assisting law enforcement agencies in identifying malicious users.

[0218] In addition, the method of this application only requires one image for recognition, while related technologies require acquiring multiple images and averaging them to enhance the PRNU pattern. Therefore, the method of this application requires less data. Furthermore, related technologies rely on image noise as a camera fingerprint, but images need to be compressed during transmission. While reducing the amount of image data, compression will process and change the details and noise patterns of the image. The PRNU pattern relies on these details and noise patterns for recognition, meaning that the PRNU pattern is sensitive to compression. This application does not use the PRNU pattern as a camera fingerprint, thus reducing the impact of compression on the final recognition result.

[0219] Figure 13 This invention illustrates a structural block diagram of a visual content recognition device provided in an exemplary embodiment of the present application. The device includes:

[0220] The acquisition module 1301 is used to acquire first visual content, which is used to characterize the visual content output when the first camera is captured, and the model of the first camera is the first model.

[0221] Feature extraction module 1302 is used to input the first visual content into the feature extraction network to obtain the first feature vector;

[0222] The determination module 1303 is used to determine the first visual content as visual content for attack when the similarity between the first feature vector and the first center vector is less than a similarity threshold. The first center vector is used to characterize the center of the feature vector of at least one visual content captured by the first type of camera.

[0223] In an optional embodiment, the determining module 1303 is further configured to obtain the first center vector obtained when training the feature extraction network if the first model belongs to the camera models in the training set of the feature extraction network.

[0224] If the similarity between the first feature vector and the first center vector is less than the similarity threshold, the first visual content is determined to be the visual content used for the attack.

[0225] In an optional embodiment, the first center vector is a portion of the weight parameters of the last fully connected layer in the feature extraction network that corresponds to the first model, and the weight parameters include center vectors corresponding to at least two camera models respectively.

[0226] In an optional embodiment, the determining module 1303 is further configured to acquire a first number of visual contents captured by the first camera when the first model does not belong to the camera models in the training set of the feature extraction network.

[0227] The first number of visual contents are input into the feature extraction network to obtain the first number of visual content features, and the first number of visual content features correspond one-to-one with the first number of visual contents.

[0228] The first center vector is obtained based on the average value of the first number of visual content features;

[0229] Based on the fact that the similarity between the first feature vector and the first center vector is less than the similarity threshold, the first visual content is determined to be the visual content used for the attack.

[0230] In an optional embodiment, the device further includes a training module 1304. The training module 1304 is used to acquire second visual content, which is visual content captured by a second camera, the second camera being a second model.

[0231] The second visual content is input into the feature extraction network to obtain the second feature vector;

[0232] Based on the similarity between the second feature vector, the second center vector, and the third center vector, a first triplet loss is obtained. The second center vector is used to characterize the center of the feature vector of at least one visual content captured by the second type of camera, and the third center vector is used to characterize the center of the feature vector of at least one visual content captured by cameras of other types besides the second type.

[0233] The feature extraction network is trained based on the first triplet loss.

[0234] In an optional embodiment, the training module 1304 is further configured to obtain a first angle based on the similarity between the second feature vector and the second center vector; and to obtain a second angle based on the similarity between the second feature vector and the third center vector; the first angle is used to characterize the angle between the second feature vector and the second center vector, and the second angle is used to characterize the angle between the second feature vector and the third center vector;

[0235] The sum of the first angle and the preset angle is used to obtain the sum of the first angles.

[0236] Based on the first angle and the second angle, the first triplet loss is obtained.

[0237] In an optional embodiment, the second visual content is cropped content on the first sample visual content; the training module 1304 is also used to obtain a third visual content, which is cropped content on the first sample visual content that is different from the second visual content.

[0238] The third visual content is input into the feature extraction network to obtain the third feature vector;

[0239] The first consistency loss is obtained based on the feature distance between the second and third feature vectors; the feature extraction network is trained based on the first consistency loss.

[0240] In an optional embodiment, the first sample visual content is one of at least two sample visual contents; the training module 1304 is also used to calculate the feature distance between the second feature vector and the third feature vector;

[0241] The first consistency loss is obtained based on the distance between at least two features corresponding to the visual content of at least two samples.

[0242] In an optional embodiment, the training module 1304 is further configured to acquire fourth visual content, which is visual content captured by a third camera, and the device number of the third camera is the first device number.

[0243] The fourth visual content is input into the feature extraction network to obtain the fourth feature vector;

[0244] Based on the similarity between the fourth feature vector, the fourth center vector, and the fifth center vector, a second triplet loss is obtained. The fourth center vector is used to characterize the center of the feature vector of at least one visual content captured by the camera with the first device number, and the fifth center vector is used to characterize the center of the feature vector of at least one visual content captured by the camera with other device numbers besides the first device number.

[0245] The feature extraction network is trained based on the second triplet loss.

[0246] In an optional embodiment, the training module 1304 is further configured to obtain a third angle based on the similarity between the fourth feature vector and the fourth center vector; and to obtain a fourth angle based on the similarity between the fourth feature vector and the fifth center vector; the third angle is used to characterize the angle between the fourth feature vector and the fourth center vector, and the fourth angle is used to characterize the angle between the fourth feature vector and the fifth center vector.

[0247] The sum of the third angle and the preset angle is used to obtain the sum of the second angle.

[0248] Based on the second angle sum and the fourth angle, the second triplet loss is obtained.

[0249] In an optional embodiment, the fourth visual content is cropped content from the second sample visual content. The training module 1304 is also configured to obtain a fifth visual content, which is cropped content from the second sample visual content that differs from the fourth visual content.

[0250] The fifth visual content is input into the feature extraction network to obtain the fifth feature vector;

[0251] The second consistency loss is obtained based on the feature distance between the fourth and fifth feature vectors; the feature extraction network is trained based on the second consistency loss.

[0252] In an optional embodiment, the second sample visual content is one of at least two sample visual contents; the training module 1304 is also used to calculate the feature distance between the fourth feature vector and the fifth feature vector.

[0253] The second consistency loss is obtained based on the distance between at least two features corresponding to the visual content of at least two samples.

[0254] In an optional embodiment, a global average pooling layer is provided before the last fully connected layer of the feature extraction network.

[0255] In summary, this application extracts features based on the first visual content and then performs matching based on the extracted first feature vector, rather than matching through image noise. This eliminates the need for a noise extraction step, which is highly computationally and storage-intensive. Therefore, the visual content recognition method provided in this application saves computational and storage resources. Furthermore, this application verifies the authenticity of the first visual content by detecting the camera model. When the first visual content is intended for attack, the method provided in this application can locate the device source of the first visual content, assisting law enforcement agencies in identifying malicious users.

[0256] In addition, the method of this application only requires one visual element for recognition, for example, only one image needs to be acquired, while related technologies require acquiring multiple images and averaging them to enhance the PRNU pattern. Therefore, the method of this application requires less data. Furthermore, related technologies rely on image noise as a camera fingerprint, but images need to undergo compression during transmission. While reducing the amount of image data, compression will process and change the details and noise patterns of the image. The PRNU pattern relies on these details and noise patterns for recognition, meaning that the PRNU pattern is sensitive to compression. However, this application does not use the PRNU pattern as a camera fingerprint, reducing the impact of compression on the final recognition result.

[0257] Figure 14 This is a schematic diagram illustrating the structure of a computer device according to an exemplary embodiment. The computer device includes... Figure 1 The computer device 1400 includes a backend server 11. The computer device 1400 includes a Central Processing Unit (CPU) 1401, a system memory 1404 including Random Access Memory (RAM) 1402 and Read-Only Memory (ROM) 1403, and a system bus 1405 connecting the system memory 1404 and the CPU 1401. The computer device 1400 also includes a basic input / output system (I / O system) 1406 to facilitate information transfer between various devices within the computer device, and a mass storage device 1407 for storing the operating system 1413, application programs 1414, and other program modules 1415.

[0258] The basic input / output system 1406 includes a display 1408 for displaying information and an input device 1409 for user input, such as a mouse or keyboard. Both the display 1408 and the input device 1409 are connected to the central processing unit 1401 via an input / output controller 1410 connected to the system bus 1405. The basic input / output system 1406 may also include the input / output controller 1410 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1410 also provides output to a display screen, printer, or other types of output devices.

[0259] The mass storage device 1407 is connected to the central processing unit 1401 via a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer device-readable media provide non-volatile storage for the computer device 1400. That is, the mass storage device 1407 may include computer device-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0260] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer device readable instructions, data structures, program modules, or other data. Computer device storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer device storage media are not limited to the above-mentioned types. The system memory 1404 and mass storage device 1407 described above can be collectively referred to as memory.

[0261] According to various embodiments of this disclosure, the computer device 1400 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1400 can be connected to the network 1411 via a network interface unit 1412 connected to the system bus 1405, or the network interface unit 1412 can be used to connect to other types of networks or remote computer device systems (not shown).

[0262] The memory also includes one or more programs stored in the memory, and the central processing unit 1401 executes the one or more programs to implement all or part of the steps of the above-described method for recognizing visual content.

[0263] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the visual content recognition method provided in the above method embodiments.

[0264] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the visual content recognition method provided in the above-described method embodiments.

[0265] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0266] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0267] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for recognizing visual content, characterized in that, The method includes: Acquire first visual content, which is used to characterize the visual content output when captured by the first camera, and the model of the first camera is the first model. The first visual content is input into the feature extraction network to obtain the first feature vector; If the similarity between the first feature vector and the first center vector is less than a similarity threshold, the first visual content is determined to be visual content for attack, and the first center vector is used to characterize the center of the feature vector of at least one visual content captured by the camera of the first model.

2. The method according to claim 1, characterized in that, The step of determining the first visual content as visual content for attack when the similarity between the first feature vector and the first center vector is less than a similarity threshold includes: If the first model belongs to the camera models in the training set of the feature extraction network, obtain the first center vector obtained when training the feature extraction network; If the similarity between the first feature vector and the first center vector is less than the similarity threshold, the first visual content is determined to be visual content used for attack.

3. The method according to claim 2, characterized in that, The first center vector is a portion of the weight parameters of the last fully connected layer in the feature extraction network that corresponds to the first model. The weight parameters include center vectors corresponding to at least two camera models.

4. The method according to claim 1, characterized in that, The step of determining the first visual content as visual content for attack when the similarity between the first feature vector and the first center vector is less than a similarity threshold includes: If the first model is not a camera model in the training set of the feature extraction network, obtain a first number of visual contents captured by the first camera. The first number of visual contents are input into the feature extraction network to obtain the first number of visual content features, and the first number of visual content features correspond one-to-one with the first number of visual contents. The first center vector is obtained based on the average value of the first number of visual content features; Based on the fact that the similarity between the first feature vector and the first center vector is less than the similarity threshold, the first visual content is determined to be visual content used for attack.

5. The method according to any one of claims 1 to 4, characterized in that, The feature extraction network was trained through the following steps: Acquire second visual content, which is visual content captured by a second camera, the second camera being a second model; The second visual content is input into the feature extraction network to obtain the second feature vector; Based on the similarity between the second feature vector, the second center vector, and the third center vector, a first triplet loss is obtained. The second center vector is used to characterize the center of the feature vector of at least one visual content captured by the second type of camera, and the third center vector is used to characterize the center of the feature vector of at least one visual content captured by cameras of other types besides the second type. The feature extraction network is trained based on the first triplet loss.

6. The method according to claim 5, characterized in that, The first triplet loss, derived based on the similarity between the second feature vector, the second center vector, and the third center vector, includes: A first angle is obtained based on the similarity between the second feature vector and the second center vector; and a second angle is obtained based on the similarity between the second feature vector and the third center vector; the first angle is used to characterize the angle between the second feature vector and the second center vector, and the second angle is used to characterize the angle between the second feature vector and the third center vector; The sum of the first angle and the preset angle is used to obtain the sum of the first angles. The first triplet loss is obtained based on the first angle and the second angle.

7. The method according to claim 5, characterized in that, The second visual content is a cropped version of the first sample visual content; the method further includes: Obtain third visual content, which is cropped content that is different from the second visual content on the first sample visual content; The third visual content is input into the feature extraction network to obtain the third feature vector; Based on the feature distance between the second feature vector and the third feature vector, a first consistency loss is obtained; based on the first consistency loss, the feature extraction network is trained.

8. The method according to claim 7, characterized in that, The first sample visual content is one of at least two sample visual contents; The first consistency loss is obtained based on the feature distance between the second feature vector and the third feature vector, including: Calculate the feature distance between the second feature vector and the third feature vector; The first consistency loss is obtained based on the distances between at least two features corresponding to the visual content of the at least two samples.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Acquire fourth visual content, which is visual content captured by a third camera, and the device number of the third camera is the first device number; The fourth visual content is input into the feature extraction network to obtain the fourth feature vector; Based on the similarity between the fourth feature vector, the fourth center vector, and the fifth center vector, a second triplet loss is obtained. The fourth center vector is used to characterize the center of the feature vector of at least one visual content captured by the camera with the first device number, and the fifth center vector is used to characterize the center of the feature vector of at least one visual content captured by the camera with other device numbers besides the first device number. The feature extraction network is trained based on the second triplet loss.

10. The method according to claim 9, characterized in that, The second triplet loss is obtained based on the similarity between the fourth feature vector, the fourth center vector, and the fifth center vector, including: A third angle is obtained based on the similarity between the fourth feature vector and the fourth center vector; and a fourth angle is obtained based on the similarity between the fourth feature vector and the fifth center vector; the third angle is used to characterize the angle between the fourth feature vector and the fourth center vector, and the fourth angle is used to characterize the angle between the fourth feature vector and the fifth center vector. Based on the sum of the third angle and the preset angle, the sum of the second angle is obtained; The second triplet loss is obtained based on the second angle and the fourth angle.

11. The method according to claim 9, characterized in that, The fourth visual content is a cropped content from the second sample visual content; the method further includes: Obtain the fifth visual content, which is a cropped content on the second sample visual content that is different from the fourth visual content; The fifth visual content is input into the feature extraction network to obtain the fifth feature vector; The second consistency loss is obtained based on the feature distance between the fourth feature vector and the fifth feature vector; the feature extraction network is trained based on the second consistency loss.

12. The method according to claim 11, characterized in that, The second sample visual content is one of at least two sample visual contents; The second consistency loss is obtained based on the feature distance between the fourth feature vector and the fifth feature vector, including: Calculate the feature distance between the fourth feature vector and the fifth feature vector; The second consistency loss is obtained based on the distances between at least two features corresponding to the visual content of the at least two samples.

13. The method according to any one of claims 1 to 12, characterized in that, A global average pooling layer is set before the last fully connected layer of the feature extraction network.

14. A visual content recognition device, characterized in that, The device includes: The acquisition module is used to acquire first visual content, which is used to characterize the visual content output when the first camera is captured, and the model of the first camera is a first model. The feature extraction module is used to input the first visual content into the feature extraction network to obtain the first feature vector; The determination module is used to determine that the first visual content is visual content for attack when the similarity between the first feature vector and the first center vector is less than a similarity threshold. The first center vector is used to characterize the center of the feature vector of at least one visual content captured by the camera of the first model.

15. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the visual content recognition method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the method for recognizing visual content as described in any one of claims 1 to 13.

17. A computer program product, characterized in that, The computer program product stores a computer program that is loaded and executed by a processor to implement the visual content recognition method as described in any one of claims 1 to 13.

Images