Virtual reality (VR) image recognition method and device, electronic device, and storage medium

Abstract

The present application provides a virtual reality (VR) image recognition method and apparatus, an electronic device, and a storage medium, wherein the virtual reality (VR) image recognition method includes: obtaining one or more target VR expanded images; cropping each target VR expanded image into multiple square sub-images; processing each square sub-image using a first attention classification model to obtain a classification result for each square sub-image, where the classification result is either a suspicious classification or a non-suspicious classification; if a suspicious classification exists in the classification result, determining that the target VR expanded image corresponding to the square sub-image belonging to the suspicious classification is a suspicious image.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a virtual reality (VR) image recognition method and device, electronic equipment, and storage medium. Background Art

[0002] With the development of internet technology, more and more websites and application platforms are disseminating information through the publication of images or videos. However, as images become more widely disseminated, the number of images containing prohibited information is also increasing. To maintain the health of the internet, websites and application platforms need to identify suspicious images, identify those containing prohibited information, and then delete them.

[0003] Traditional suspicious image identification methods rely on manual recognition, but due to the huge number of images published on the Internet, manual recognition is time-consuming and labor-intensive, and cannot guarantee the accuracy and reliability of suspicious image recognition, which is not conducive to the healthy development of the Internet industry. Summary of the Invention

[0004] To solve the above technical problems, the embodiments of the present invention provide a virtual reality (VR) image recognition method and device, electronic device, and storage medium, which can automatically identify suspicious images in VR images and improve the efficiency of suspicious image recognition.

[0005] The present invention provides a method for virtual reality (VR) image recognition, including:

[0006] Obtain one or more target VR expansion images;

[0007] Crop each target VR expansion image into multiple square sub-images;

[0008] Each square sub-image is processed using the first attention classification model to obtain a classification result for each square sub-image, which is either a suspicious classification or a non-suspicious classification;

[0009] If there is a suspicious classification in the classification result, the target VR expansion image corresponding to the square sub-image belonging to the suspicious classification is determined to be a suspicious image.

[0010] The embodiment of the present application provides a virtual reality (VR) image recognition device, comprising:

[0011] A target acquisition unit, configured to acquire one or more target VR expansion images;

[0012] A cropping unit, used to crop each target VR expansion image into multiple square sub-images;

[0013] a model recognition unit, configured to process each square sub-image using the first attention classification model to obtain a classification result for each square sub-image, wherein the classification result is a suspicious classification or a non-suspicious classification;

[0014] The determination unit is used to determine whether the target VR expansion image is a suspicious image.

[0015] An embodiment of the present application provides an electronic device, including:

[0016] a memory for storing executable instructions;

[0017] The processor is configured to implement the virtual reality (VR) image recognition method when executing the executable instructions stored in the memory.

[0018] An embodiment of the present application provides a storage medium storing executable instructions. When the executable instructions are executed by at least one processor, the virtual reality (VR) image recognition method is implemented.

[0019] The present application provides a virtual reality (VR) image recognition method, which includes obtaining one or more target VR expanded images; cropping each target VR expanded image into multiple square sub-images; processing each square sub-image using a first attention classification model to obtain a classification result for each square sub-image, wherein the classification result is either a suspicious classification or a non-suspicious classification; and if a suspicious classification exists in the classification result, determining that the target VR expanded image corresponding to the square sub-image belonging to the suspicious classification is a suspicious image.

[0020] In the embodiment of the present application, one or more target VR expanded images are obtained; each target VR expanded image is cropped into multiple square sub-images. This allows the first attention classification model to be used to process each square sub-image, obtaining a classification result for each square sub-image, and then identifying whether the target VR expanded image is a suspicious image. This achieves automatic identification of suspicious images in VR images and improves the efficiency of suspicious image identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a virtual reality (VR) image recognition method according to an embodiment of the present application;

[0022] Figure 2 A schematic diagram of the structure of the SENet model provided in the embodiment of the present application;

[0023] Figure 3 A flowchart of inputting a preprocessed training set into a second attention classification model based on the SENet model for training according to an embodiment of the present application;

[0024] Figure 4A flowchart of a virtual reality (VR) image recognition method provided in an embodiment of the present application;

[0025] Figure 5 A flowchart of a method for constructing a training set, a test set, and a validation set provided in an embodiment of the present application;

[0026] Figure 6 A flowchart of a method for preprocessing the training set and the validation set provided in an application embodiment of the present application;

[0027] Figure 7 A flowchart of a method for constructing a second attention classification model provided in an embodiment of the present application;

[0028] Figure 8 A flowchart of a method for correcting distortion of a test set provided in an embodiment of the present application;

[0029] Figure 9 A schematic diagram of the structure of a virtual reality (VR) image recognition device provided in an embodiment of the present application;

[0030] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will describe the technical embodiments of the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] It should be noted that in the embodiments of the present application, the term "and / or" is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the embodiments of the present application, the character " / " generally indicates that the associated objects are in an "or" relationship.

[0033] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between the two, or an association relationship between the two, or a relationship between indication and being indicated, configuration and being configured, etc.

[0034] To facilitate understanding of the technical embodiments of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following related technologies can be arbitrarily combined with the technical embodiments of the embodiments of the present application as optional embodiments, and they all fall within the protection scope of the embodiments of the present application.

[0035] Suspicious image recognition involves identifying whether a target image contains prohibited information, thereby determining whether the target image is suspicious. With the development of models, the types of target images are increasing, and the demand for target image recognition is also increasing. For example, live video streaming, e-commerce malls, social forums, and other platforms all contain diverse image information. Using only manual recognition methods will undoubtedly seriously affect image recognition efficiency.

[0036] In recent years, with the rapid development of convolutional neural models, convolutional neural models have achieved good results in image detection, recognition, and classification. The basic steps for using convolutional neural models to identify suspicious images are as follows: the RGB information of the target image is input into the image recognition model based on the convolutional neural model for detection. The image recognition model extracts features of the target image through the convolutional neural model, obtains key information and inputs it into the image classification model based on the convolutional neural model to determine whether the target image contains prohibited information. If prohibited information is present, a secondary verification is performed through manual recognition, which greatly reduces the complexity of manual recognition and improves the efficiency of suspicious image recognition. For example:

[0037] 1) The current live broadcast volume is huge, resulting in an even greater amount of image information contained in live broadcast videos. It is extremely difficult to identify suspicious images through manual recognition alone. However, using convolutional neural models for suspicious image recognition can achieve parallel recognition of multiple contents and high-precision identification of prohibited content, greatly improving the efficiency of suspicious image recognition.

[0038] 2) E-commerce malls have become the preferred choice for residents when shopping. By using convolutional neural models to identify suspicious images, identify image interaction information uploaded by merchants and users, and efficiently issue warnings for non-compliant images, it can effectively improve image recognition efficiency and reduce the risk of business violations.

[0039] 3) Application platforms such as social forums and user-generated content (UCG) websites contain complex image information. Using convolutional neural models to identify suspicious images can effectively improve the quality of the image content contained in the application platforms.

[0040] Furthermore, taking suspicious image recognition as an example, suspicious image recognition refers to identifying whether there are suspicious elements in the image, so as to achieve the purpose of quickly screening prohibited information.

[0041] Suspicious image recognition is mainly divided into the following identification methods:

[0042] 1) Construct a convolutional neural model to obtain the features of the target image, input the features of the target image into the classification model, and finally output the classification label vector of the target image, thereby achieving the purpose of identifying the category label of the target image.

[0043] 2) Based on target detection, a training dataset is constructed by annotating category labels, and a convolutional neural model is trained to complete target detection and type recognition, thereby accurately identifying suspicious elements in the target image.

[0044] The following introduces several suspicious identification implementation examples:

[0045] Example A: This example combines target detection and image classification to perform secondary detection on the image, that is, firstly through suspicious element detection to determine whether the image contains suspicious elements, and then through scene recognition to determine whether the image contains suspicious scenes. If it is determined that the image contains suspicious elements and suspicious scenes, it is determined that there is suspicious content in the scene.

[0046] Example B: This example obtains an image classification probability vector through an image classification model, that is, standardizes the image to a size of 224*224, and obtains a label probability vector through a labeling model to characterize the labeling information of the image. All labels whose probability vector is greater than the threshold are the labeling results of the image.

[0047] The defects of the above embodiment are as follows:

[0048] For Example A, first, since the training set data of suspicious images is difficult to obtain, if suspicious elements are to be detected in the image, a large amount of data labeling work is required, which requires huge manpower and time costs; second, suspicious element detection and suspicious scene recognition must be completed before it can be determined whether the image is a suspicious image. The judgment conditions are relatively harsh. If the judgment accuracy of one step is low, it will have a great impact on the recognition result.

[0049] For Example B, first, this example obtains a label probability vector by training a labeling model, and its input is a standardized square graph, which has a limited scope of application.

[0050] In particular, with the development of virtual reality (VR) technology, when viewers wear VR glasses and other devices to watch VR videos and VR panoramas, they are presented as 360-degree panoramas, in which viewers can interact with virtual scenes or objects to obtain an immersive experience. As a result, the content, quantity, and types of VR images on the Internet (including VR panoramas, VR videos, and VR expanded images, where VR videos are formed by playing VR panoramas at a certain frame rate, and VR expanded images are formed by expanding VR panoramas) are becoming increasingly diverse. When VR images are identified using only conventional suspicious image recognition methods, such as the methods of Examples A and B, in addition to the above-mentioned defects, the following defects also exist:

[0051] 1) VR expanded image distortion problem: VR panoramas are 360° panoramic images. Due to the nonlinear factors of the imaging model, if the VR panorama is expanded to a 2D plane to form a VR expanded image, the image will be significantly distorted, which will greatly interfere with the subsequent image feature extraction and recognition.

[0052] 2) The problem of loss of accuracy of VR expansion images: The image recognition model based on the convolutional neural model needs to fill the input image into a square image and then adjust it to a fixed size, such as 224*224, to be sent to the image classification model for recognition and classification; since the length and width of VR expansion images (such as cylindrical expansion images and perspective expansion images) are not consistent, and they have rich pixel accuracy, if the VR expansion image is filled into a square image and then compressed, the original VR expansion image needs to be filled with invalid pixels, which will introduce a large amount of redundant interference information for the convolution operation in the image classification model, and compressing the square image into a smaller pixel representation will result in the loss of more effective feature information, which will have a negative impact on the accuracy of image classification and recognition.

[0053] 3) Detail recognition problem in VR expanded images: Due to the panoramic surround characteristics of VR panoramas, VR panoramas often contain complex information, and the effective features of the image are only highlighted by some key features. If these key features only occupy a small proportion in the VR expanded image, the image classification model may ignore the key feature information when recognizing the image, resulting in incorrect judgment.

[0054] In order to solve at least some of the above-mentioned defects, the following technical embodiments of the embodiments of the present application are proposed.

[0055] In order to enable a more detailed understanding of the features and technical content of the present application, the implementation of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the present application. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0056] This embodiment provides a virtual reality (VR) image recognition method. Figure 1 This is a flow chart of the virtual reality (VR) image recognition method provided in this embodiment. Figure 1 As shown, the virtual reality VR image recognition method includes the following steps:

[0057] Step 101: Obtain one or more target VR expansion images;

[0058] Step 102: Crop each target VR expansion image into multiple square sub-images;

[0059] Step 103: using the first attention classification model to process each square sub-image to obtain a classification result for each square sub-image, where the classification result is a suspicious classification or a non-suspicious classification;

[0060] Step 104: If there is a suspicious category in the classification result, the target VR expansion image corresponding to the square sub-image belonging to the suspicious category is determined to be a suspicious image; if there is no suspicious category in the classification result, all target VR expansion images are determined to be non-suspicious images.

[0061] Explanation of the terms appearing in the above embodiments:

[0062] VR expanded image: A VR panorama (such as a circular panorama) expanded onto a 2D plane.

[0063] Square sub-image: A square sub-image cropped from the target VR unfolded image.

[0064] Suspicious images: VR expansion images containing prohibited information.

[0065] Since the first attention classification model is a model obtained through machine learning training, the accuracy of suspicious image recognition cannot be 100%. Therefore, after the first attention classification model determines that the target VR expansion image is a suspicious image, the target VR expansion image needs to be manually reviewed to improve the accuracy of suspicious image recognition and ensure the reliability of the suspicious image recognition results.

[0066] Based on this, in one embodiment, the method further includes:

[0067] If the target VR expanded image is determined to be a suspicious image, the suspicious area is marked at the corresponding position of the target VR expanded image and manually reviewed;

[0068] The suspicious area refers to the suspicious area of ​​the square subgraph corresponding to the suspicious classification in the classification result.

[0069] In the above embodiment, if one or more suspicious classifications are found in the classification results of a square sub-image cropped from a target VR expansion image, the suspicious areas of the square sub-image corresponding to each suspicious classification need to be marked on the corresponding target VR expansion image. Through manual review, as long as it is determined that one of the suspicious areas contains prohibited information, the target VR expansion image is determined to be a suspicious image. In the above embodiment, before using the attention classification model, when the structure of the attention classification model is determined, it is also necessary to train the attention classification model using a large number of training sets and update the parameters of the attention classification model to ensure that it obtains a higher recognition accuracy in specific application scenarios, such as the VR image recognition application scenario in the embodiment of the present application. Therefore, the attention classification model also needs to be trained in the above embodiment.

[0070] Based on this, in one embodiment, the method further includes: training the second attention classification model to obtain the first attention classification model, where the first attention classification model refers to the trained second attention classification model.

[0071] In the above embodiment, when training the second attention classification model, the second attention classification model can be first trained using the training set, and the parameters of the second attention classification model can be updated using the loss function, and then the second attention classification model with the best effect can be selected using the validation set. It should be noted that the attention classification model is also a type of deep learning model. The deep learning model has format requirements for the input object, so before using the training set and the validation set, the training set and the validation set must be preprocessed.

[0072] Based on this, in one embodiment, the training of the second attention classification model includes:

[0073] Construct training and validation sets;

[0074] Preprocess the training set and validation set;

[0075] The preprocessed training set is input into the second attention classification model for training, and the parameters of the second attention classification model are updated using the loss function;

[0076] The model with the best effect on the preprocessed validation set is selected as the second attention classification model.

[0077] In the above embodiment, since the function of the first attention classification model is to identify whether the square sub-image belongs to the suspicious category or the non-suspicious category, in order to achieve a better recognition effect, the construction of the training set and the validation set is very important.

[0078] Based on this, in one embodiment, constructing a training set and a validation set includes:

[0079] Based on the suspicious categories in the classification results, use the keywords corresponding to the suspicious categories to crawl multiple suspicious category images, and mark each suspicious category image with a suspicious category label;

[0080] Based on the non-suspicious classification in the classification result, multiple non-suspicious classification images are obtained from an open source dataset based on image classification, and each non-suspicious classification image is labeled with a non-suspicious classification label; specifically, the open source dataset can be downloaded from the Internet, and the image classification can select the non-suspicious classification, that is, a certain number of images are extracted from the non-suspicious classification in the open source dataset as non-suspicious classification images;

[0081] A training set and a validation set are constructed based on suspicious and non-suspicious classified images.

[0082] In the above embodiment, for a training set with a larger data set, when the convolutional neural model performs gradient descent on the loss function and parameters, it is necessary to perform gradient descent on all samples for a large data set, which is impossible to achieve under limited memory conditions. If the training set is batched, then each gradient descent is only performed on one batch of data. In this way, the amount of data input into the convolutional neural model each time is small, and training can be performed under limited memory conditions.

[0083] Furthermore, due to the necessity of batches, the training of convolutional neural models is based on batches and backpropagated once. If a batch contains all the training samples in the training set, then it makes no difference whether the training set is shuffled or not, because during backpropagation, the convolutional neural model will calculate the loss of each sample in the batch and then find the average value. Therefore, whether the training set is shuffled, that is, the order, has no effect on the average value of the sample loss. However, when a batch only contains part of the training samples in the training set, each batch is required to be as close to the true probability distribution of the samples as possible. The samples after random shuffling are closer to the true probability distribution than the sorted samples, because there is an extra layer of correlation between the sorted samples, which will show different probability distributions to a certain extent. That is, the convolutional neural model after sample sorting learns different data distributions each time it is trained, which is not conducive to model convergence.

[0084] Based on this, in one embodiment, the preprocessing of the training set and the validation set includes:

[0085] Shuffle and batch the training set to obtain multiple training subsets;

[0086] Fill each training subset and validation set with images into square graphs;

[0087] The square graphs in each training subset and validation set are compressed to obtain a preprocessed training set and a preprocessed validation set, where the preprocessed training set contains multiple preprocessed training subsets.

[0088] In the above embodiment, in the convolutional neural model, the input image is usually required to be square in size, while the images in the training subset and the validation set may be rectangular in size. If the rectangular image is directly stretched or compressed, it will cause the image to be deformed, which is not conducive to the training of the convolutional neural model. Therefore, in the embodiment of the present application, a filling method is adopted to fill the images in each training subset and the validation set into square images. The specific method is as follows:

[0089] Use the input image length or width to do geometric scaling;

[0090] Use the copyMakeBorder function in OpenCV to fill the borders of the remaining pixels. The grayscale value of the pixels used for border filling is usually 128.

[0091] Furthermore, since in the embodiment of the present application, the training set is divided into multiple batches after shuffle, when using any batch to train the second attention classification model, it is necessary to use the loss function to update the parameters of the second attention classification model.

[0092] Based on this, in one embodiment, inputting the preprocessed training set into the second attention classification model for training, and updating the parameters of the second attention classification model using the loss function, includes:

[0093] Each preprocessed training subset is input into the second attention classification model for training, and the parameters of the second attention classification model are updated using the loss function.

[0094] When training the second attention classification model, standardizing the training set and validation set helps avoid model overfitting and speeds up model fitting.

[0095] Based on this, in one embodiment, the training set is shuffled and processed in batches to obtain multiple training subsets, including:

[0096] Standardize the images in the training set and validation set to obtain a standardized training set and a standardized validation set;

[0097] The standardized training set is shuffled and batched to obtain multiple training subsets.

[0098] In the above embodiments, the standardization process generally includes the following two embodiments:

[0099] 1. Unify the training set and validation set for standardization;

[0100] 2. First standardize the training set, and then use the standardization rules of the training set to standardize the validation set data.

[0101] In the embodiment of the present application, the second method is selected, that is, first normalizing the training set and then normalizing the validation set data using the normalization rules of the training set. The reason is that:

[0102] The first method standardizes all data in the training set and validation set. It is suitable for use when all data are available, but not for adding data to an already trained model.

[0103] The second method is to extract the data normalization rules of the training set and then use them on the validation set. This method can standardize the input data according to the training set rules after the second attention classification model is trained, and the trained second attention classification model can be used on the input data later.

[0104] The intuitive comparison between the first method and the second method is as follows:

[0105] Suppose there is a set of data with a maximum value of 100 and a minimum value of -100. After the training set and validation set are unified and normalized, the maximum value of the data becomes 1 and the minimum value becomes 0.1. If a batch of real-time data arrives after the second-attention classification model is trained, the real-time data will not be able to obtain the normalization rules of the training set. Moreover, the real-time data will lose its inherent characteristics when normalized using its own rules. For example, if a batch of data A is passed in with a maximum value of 200, according to the standardization rules of A, the maximum value will become 1, which is consistent with the normalized data of the training set maximum value of 100. Therefore, the second-attention classification model will lose the characteristics of batch A.

[0106] If the validation set is also standardized according to the rules of the training set, a batch of data A is now passed in with a maximum value of 200. At this time, after standardization according to the rules of the training set, the maximum value will become 1.5, and the characteristics of this batch of data A will be correctly fed back.

[0107] Based on this, in one embodiment, the images in the training set and the validation set are normalized to obtain a normalized training set and a normalized validation set, including:

[0108] The training set is standardized using the z-score standardization method to obtain a standardized training set;

[0109] Based on the mean and variance of the training set, the validation set is standardized using the z-score standardization method to obtain the standardized validation set.

[0110] In the above embodiment, the main purpose of the Z-Score normalization method is to convert data of different magnitudes into the same magnitude and measure them uniformly with the calculated Z-Score value to ensure comparability between data. It can be expressed mathematically as follows: In the embodiment of the present application, x R,G,B It represents the value of the R channel, G channel, or B channel of any pixel in the training set or validation set, μ represents the mean of the values ​​of the R channel, G channel, or B channel corresponding to all pixels in the training set, δ represents the standard deviation of the values ​​of the R channel, G channel, or B channel corresponding to all pixels in the training set, and the variance is the arithmetic square root of the standard deviation.

[0111] In the above embodiment, a successfully trained second attention classification model requires updating a large number of parameters. These parameters require a large number of training sets for training to work correctly. In reality, there are not so many pictures that can be used in the training set, but we can improve the generalization ability of the model by increasing the amount of training data. Among the existing methods, there are two main methods to increase the amount of training data: one method is to obtain new data, which is more troublesome and requires a lot of cost; the second method is to enhance the data, that is, to use existing data for operations such as flipping, translating or rotating to create more data to make the model have better generalization effect.

[0112] Based on this, in one embodiment, before compressing the quad graphs in each training subset and validation set, the method further includes:

[0113] Perform data augmentation on the quad images in each training subset using at least one of the following methods: flipping, translating, rotating, scaling, cropping, or introducing noise.

[0114] Furthermore, convolutional neural networks have achieved significant breakthroughs in image recognition. The convolution kernel, at the core of these models, is an information aggregator used to aggregate spatial and channel-wise information within a local receptive field. Convolutional neural networks consist of a series of convolutional, nonlinear, and downsampling layers to capture image features from a global receptive field and subsequently describe the image.

[0115] Squeeze-and-Excitation Networks (SENet) approach this from a different perspective. By adding squeeze and excitation operations, they explicitly model the interdependencies between feature channels, rather than introducing new spatial or dimensional information. Specifically, they automatically learn the importance of each feature channel and employ a feature recalibration strategy to increase the weight of useful features based on their importance, while suppressing the weight of features that are unimportant to the task at hand.

[0116] Based on this, Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of the SENet model provided in the embodiment of the present application. Figure 2 The SENet model shown is a structural diagram of the SE-ResNet model formed by inserting the SE module (Squeeze-and-Excitation Blook) into the ResNet model.

[0117] Based on this, in one embodiment, inputting the preprocessed training set into the second attention classification model for training includes:

[0118] The preprocessed training set is input into the second attention classification model based on the SENet model for training.

[0119] In the above embodiment, if Figure 3 As shown, Figure 3 A flowchart of inputting a preprocessed training set into a second attention classification model based on the SENet model for training is provided in an embodiment of the present application; the inputting a preprocessed training set into a second attention classification model based on the SENet model for training includes:

[0120] Step 301: Use the residual layer of the ResNet model to extract the output feature Z of the preprocessed training set conv , whose dimension is (c, h, w); where c represents the number of convolution kernels, i.e. the output feature Z conv The number of feature channels, or the residual layer of the ResNet model outputs c feature maps of size h*w;

[0121] Step 302: Output feature Z conv Perform global average pooling operation to obtain the output feature Z conv The feature distribution W of each feature channel mean, whose dimension is (c,1,1); the global average pooling operation can represent the output feature Z conv The numerical distribution of the c feature maps of size h*w in , that is, the feature distribution W of each feature channel mean , that is, global information,

[0122] Step 303: Distribute the features of each feature channel W mean Input the first fully connected layer (FC) to obtain the feature distribution W1*W after dimensionality reduction mean ; Among them, the dimension of the first fully connected layer W1 is r is a scaling parameter, whose purpose is to reduce the number of feature channels and thus reduce the amount of computation. After the dimensionality reduction operation of the first fully connected layer, the output is the reduced-dimensional feature distribution W1*W mean , whose dimensions are

[0123] Step 304: Distribute the reduced dimension features W1*W mean Enter the RELU activation function for activation, and get the activated feature distribution RELU (W1*W mean ), its dimension remains unchanged; the activation function is used to add nonlinear factors, improve the expressive power of the convolutional neural model, and solve problems that the linear model cannot solve;

[0124] Step 305: Distribute the activated features into RELU (W1*W mean ) is sent to the second fully connected layer W2 to obtain the feature distribution W2*RELU(W1*W mean ); the dimension of the second fully connected layer W2 is Therefore, the feature distribution after dimension restoration is W2*RELU(W1*W mean ) has a dimension of (c,1,1);

[0125] Step 306: Restore the dimension of the feature distribution W2*RELU(W1*W mean ) is sent to the Sigmoid activation function to obtain the output feature Z conv The weight W of each feature channel full , where W full =Sigmoid(W2*RELU(W1*W mean )), W full Used to represent Z conv The weight of each feature map of size h*w in Z conv The weight of each feature channel, and W fullIt is learned through the first fully connected layer W1, the second fully connected layer W2, the RELU activation function layer and the Sigmoid activation function layer, so it can be used for end-to-end training of the ResNet model; it can also be seen that the role of the first fully connected layer W1 and the second fully connected layer W2 is to fuse the output feature Z conv The information of the feature map of each feature channel is obtained because each Squeeze (feature compression) is actually operated on the feature map of a certain feature channel;

[0126] Step 307: Using output feature Z conv The weight W of each feature channel full For the output feature Z conv Perform weighted operation to obtain weighted output features in,

[0127] Step 308: The weighted output features Input the fully connected layer and classification layer of the SE-ResNet model, and output the classification results of each image in the training set; the activation function of the classification layer is softmax, and the classification result in the embodiment of the present application indicates whether the target VR expansion image is a suspicious classification or a non-suspicious classification.

[0128] In the second attention classification model, the loss of each training subset is the average of the errors between the predicted value and the true value of each sample in each training subset. In this embodiment of the present application, the cross-entropy loss function is used to represent the loss of each training subset, that is, the cross-entropy loss function is used as the loss function of the second attention classification model. Wherein, cross-entropy = relative entropy + information entropy.

[0129] The definition of relative entropy (KL divergence) is: if there are two separate probability distributions P(x) and Q(x) for the same random variable X, then we can use KL divergence to measure the difference between the two probability distributions, which can be expressed mathematically as follows: Among them, p(x i ) represents the true distribution of the sample, q(x i ) represents the sample distribution predicted by the model, i represents the i-th sample, and n represents the total number of samples; the mathematical expression of relative entropy can also be rewritten as:

[0130]

[0131] Among them, -Ηp(x) represents information entropy, That is the cross entropy H(p,q), that is

[0132] When training a machine learning model, the input data and labels are often already determined, so the true probability distribution P(x) is also determined, so the information entropy is a constant here. Since the value of the KL divergence represents the difference between the true probability distribution P(x) and the predicted probability distribution Q(x), the smaller the value of the KL divergence, the better the prediction result, so it is necessary to minimize the KL divergence, and the cross entropy is equal to the KL divergence plus a constant (information entropy). From the mathematical expression, it can be seen that the cross entropy is easier to calculate than the KL divergence. Therefore, in the embodiment of the present application, the cross entropy loss function is used as the loss function of the second attention classification model.

[0133] Furthermore, the cross entropy loss function is used to derive the adjustable parameters, and the parameter update direction is taken as the negative derivative direction. This ensures that the loss function is adjusted in the decreasing direction until the loss function is minimized, so that the difference between the true probability distribution P(x) and the predicted probability distribution Q(x) in the sample set is minimized, that is, the model achieves the optimal effect, and then the parameters of the second attention classification model are updated.

[0134] Furthermore, the stochastic gradient descent algorithm, which is a commonly used method for minimizing the loss function, and the Adam algorithm are a first-order optimization algorithm that can replace the traditional stochastic gradient descent algorithm. It can iteratively update the weights of the neural network based on the training data, and has the advantages of high computational efficiency, low memory requirements, and invariance to diagonal gradient scaling.

[0135] Based on this, in one embodiment, the updating of the parameters of the second attention classification model using the loss function includes:

[0136] Use the Adam algorithm to minimize the cross entropy loss function;

[0137] The cross entropy loss function is used to update the parameters of the second attention classification model.

[0138] In an embodiment of the present application, the VR image published on the Internet may also exist in the form of a VR video. Based on this, in one embodiment, obtaining one or more target VR expansion images includes:

[0139] Extract one or more VR images from the target VR video as the target VR image;

[0140] Expand each target VR image to obtain one or more target VR expanded images.

[0141] In the above embodiment, as long as any target VR expansion image is determined to be a suspicious image, the corresponding target VR video can be determined to be a suspicious image.

[0142] Furthermore, the target VR image extracted from the target VR video is presented in the format of a VR panorama. If it is to be recognized by the first attention classification model, the target VR image needs to be unfolded to obtain the target VR unfolded image. For example, to obtain the target VR unfolded image by unfolding the annular VR panorama, the annular VR panorama is first projected onto the cylindrical surface using the principle of inverse ray tracing, and then the cylindrical surface is unfolded to obtain the target VR unfolded image. However, since the part of the annular VR panorama close to the inner diameter is compressed more, the target VR unfolded image obtained after unfolding will inevitably produce a certain degree of distortion, which is not conducive to the recognition of the first attention classification model. If accurate and reliable recognition results are to be obtained, the target VR unfolded image needs to be distorted.

[0143] Furthermore, the target VR expansion image obtained by cylindrical expansion is usually a rectangular image with different lengths and widths. However, the first attention classification model requires the input image to be square. Usually, the methods used to convert rectangular images into square images are stretching and filling.

[0144] The stretching method is to stretch or compress the image in the length or width direction until the image is converted to a square size. However, if the rectangular image is stretched or compressed directly, the image will be deformed or even lose features, which is not conducive to the recognition of the first attention classification model.

[0145] The filling method is to scale the length or width of the rectangular image proportionally, and then fill the remaining pixels with borders. The grayscale value of the pixels used for border filling is usually 128; however, filling the borders of the remaining pixels will introduce a large number of redundant features on the one hand, and reduce the proportion of useful features on the other hand, which is not conducive to the recognition of the first attention classification model.

[0146] Therefore, in the embodiment of the present application, a rectangular image is cropped into multiple square sub-images using a square-sized sliding window, which not only increases the proportion of useful features in the square sub-images, but also does not introduce new redundant features.

[0147] Based on this, in one embodiment, the step of cutting each target VR expansion image into a plurality of square sub-images includes:

[0148] Perform distortion correction on each target VR expansion image to obtain a corrected target VR expansion image;

[0149] Use a sliding window to crop each rectified target VR image into multiple square sub-images;

[0150] Compress each square subgraph to obtain a compressed square subgraph.

[0151] Based on this, in one embodiment, the distortion correction of the target VR expansion image includes:

[0152] Based on the coordinates of the center position of the target VR expansion image and the coordinates of any pixel point in the target VR expansion image, the distance between any pixel point and the center position in the target VR expansion image is obtained; wherein the coordinates of the center position of the target VR expansion image are (c x ,c y ), the coordinates of any pixel point in the target VR expansion map are (x, y), then according to the point-to-point distance formula, the distance between any pixel point and the center position in the target VR expansion map is

[0153] Based on the distance between any pixel point and the center position in the target VR expansion image and the distortion coefficient, a distortion correction coefficient is obtained. The distortion coefficient includes at least one of the following: pincushion distortion coefficient, barrel distortion coefficient, radial distortion coefficient, and linear scaling coefficient. The mathematical expression of the distortion correction coefficient is:

[0154] λ=k1r+k2r 2 +k3r 3 +k4r 4 ;

[0155] Where λ represents the distortion correction coefficient, k1 represents the linear scaling coefficient, which is used to describe the linear scaling degree of the target VR expansion image, k2 is the barrel distortion coefficient (calculated by the polynomial correction algorithm), k2 is a negative value, k3 is the radial distortion coefficient, which is generally 0, and k4 is the pincushion distortion coefficient, which is a positive value;

[0156] Furthermore, k1=1-k2-k3-k4;

[0157] Based on the coordinates of the center position of the target VR expansion image and the distortion correction coefficient, the position of each pixel point in the target VR expansion image after correction is obtained; among which, the position coordinates of the corresponding pixel point in the corrected target VR expansion image are

[0158] Based on the position of each pixel point in the target VR expansion image in the corrected target VR expansion image, each pixel point in the target VR expansion image is mapped to the corresponding position of the corrected target VR expansion image to obtain the corrected target VR expansion image.

[0159] Based on this, in one embodiment, the method of using a sliding window to crop the corrected target VR expanded image into multiple square sub-images includes:

[0160] The corrected target VR unfolded image is cropped using a square sliding window. The sliding step of the sliding window is the first sliding step, and the sliding direction of the sliding window is along the length direction of the corrected target VR unfolded image. Denote the size of the corrected target VR unfolded image as (w, h), the size of the sliding window as (w, w), and the first sliding step as l1.

[0161] If, after one or more croppings, the remaining length of the corrected target VR unfolded image is less than the first sliding step, the sliding window uses the remaining length as the sliding step to perform the last cropping on the corrected target VR unfolded image, obtaining multiple square sub-images. Denote the remaining length as l2, where l2 < l1, then the number of the obtained square sub-images is

[0162] An embodiment of the present application provides a virtual reality (VR) image recognition method. Specifically, one or more target VR unfolded images are obtained; each target VR unfolded image is cropped into multiple square sub-images; a first attention classification model is used to process each square sub-image to obtain a classification result for each square sub-image, and the classification result is a suspicious classification or a non-suspicious classification; if there is a suspicious classification in the classification results, it is determined that the target VR unfolded image corresponding to the square sub-image belonging to the suspicious classification is a suspicious image.

[0163] In an embodiment of the present application, one or more target VR unfolded images are obtained; each target VR unfolded image is cropped into multiple square sub-images. In this way, a first attention classification model can be used to process each square sub-image to obtain a classification result for each square sub-image, and further identify whether the target VR unfolded image is a suspicious image, realizing the automatic recognition of suspicious images in VR images and improving the efficiency of suspicious image recognition.

[0164] The following further describes the present application in combination with application embodiments: <00,00429> Figure 4 is a schematic flowchart of the virtual reality (VR) image recognition method provided by the application embodiment of the present application. As Figure 4 shown, the application embodiment of the present application provides a virtual reality (VR) image recognition method for identifying whether a target VR unfolded image is a suspicious image.

[0166] As Figure 4 shown, the virtual reality (VR) image recognition method provided by the application embodiment of the present application includes:

[0167] Step 401: Construct a data set. Specifically, construct a training set, a test set, and a validation set.

[0168] Figure 5A flow chart of the method for constructing a training set, a test set, and a validation set provided in the application embodiment of the present application is shown in FIG. Figure 5 As shown, the method for constructing a training set, a test set, and a validation set provided by the application embodiment of the present application includes:

[0169] Step 501: crawl multiple suspicious images on the Internet based on relevant keywords, and label each suspicious image with a suspicious classification label;

[0170] Step 502: Download an open source dataset for image classification, randomly extract multiple non-suspicious images from the non-suspicious category of the open source dataset, and label each non-suspicious image with a non-suspicious category label;

[0171] Step 503: constructing a training set and a validation set based on the suspicious images and the non-suspicious images;

[0172] Step 504: extract one or more VR image frames from the target VR video as target VR images, expand the target VR images to obtain one or more target VR expansion images, and construct a test set based on the one or more target VR expansion images.

[0173] Step 402: Preprocess the training set and the validation set.

[0174] Figure 6 A flow chart of the method for preprocessing the training set and the validation set provided in the application embodiment of the present application is as follows: Figure 6 As shown, the method for preprocessing the training set and the validation set provided by the application embodiment of the present application includes:

[0175] Step 601: Standardize the training set using the z-score standardization method to obtain a standardized training set; standardize the validation set using the z-score standardization method based on the mean and variance of the training set to obtain a standardized validation set;

[0176] Step 602: Shuffle and batch the standardized training set to obtain multiple training subsets, each of which contains τ samples.

[0177] Step 603: performing data augmentation processing on the square graphs in each training subset using at least one of the following methods: flipping, translating, rotating, scaling, cropping, and introducing noise;

[0178] Step 604: Fill the images in each training subset and validation set into a square image; compress the square images in each training subset and validation set to a size of (m, m) to obtain a preprocessed training set and a preprocessed validation set, wherein the preprocessed training set contains multiple preprocessed training subsets.

[0179] Step 403: Construct a second attention classification model.

[0180] Figure 7 A flow chart of a method for constructing a second attention classification model provided in an embodiment of the present application is shown as follows: Figure 7 As shown, the method for constructing the second attention classification model provided by the application embodiment of the present application includes:

[0181] Step 701: define the dimension of each training subset P as (τ, m, m, 3), where 3 represents the RGB feature dimension;

[0182] Step 702: Input the training subset into a pre-trained ResNet model (e.g., ResNet101 model), and use the residual layer of the ResNet model to extract the initial features Y of the training subset. The dimensions of the initial features Y are (τ, m, m, c1), where τ is the number of square graphs in each training subset, m is the side length of the square graph, and c1 is the feature dimension of the initial features Y.

[0183] Step 703: Input the initial feature Y into the multi-layer convolution model to obtain the output feature Z conv , whose dimension is (τ,m,m,c2); where c2 represents the output feature Z conv characteristic dimensions;

[0184] Step 704: Output feature Z conv Perform global average pooling operation to obtain output feature Z conv The feature distribution W of each feature channel mean , W mean The dimension is (τ,1,1,c2), W mean It can be expressed mathematically as:

[0185]

[0186] Where i represents the i-th pixel in the height direction, and j represents the j-th pixel in the width direction;

[0187] Step 705: W mean Input the first fully connected layer W1 to obtain the feature distribution W1*W after dimensionality reduction mean , whose dimensions are Among them, the dimension of the first fully connected layer W1 is r is the scaling parameter; W1*W mean Enter the RELU activation function for activation, and get the activated feature distribution RELU (W1*W mean ), whose dimensions are The activated feature distribution RELU(W1*W mean) is sent to the second fully connected layer W2 to obtain the feature distribution W2*RELU(W1*W mean ), whose dimension is (τ,1,1,c2), where the dimension of the second fully connected layer W2 is Change W2*RELU(W1*W mean ) is sent to the Sigmoid activation function to obtain the output feature Z conv The weight W of each feature channel full , whose dimensions are (τ,1,1,c2), W full It can be expressed mathematically as:

[0188] W full =Sigmoid(W2*RELU(W1*W mean ));

[0189] Step 706: Using W full For the output feature Z conv Perform weighted operation to obtain weighted output features The mathematical expression is:

[0190] Step 707: The weighted output features Input the fully connected layer and the classification layer, and output the classification results of each image in the training subset, which are classified as suspicious or non-suspicious. The activation function of the classification layer is softmax.

[0191] Step 708: Construct a cross entropy loss function as the loss function of the second attention model.

[0192] Step 404: Training a second attention classification model, including:

[0193] Use the Adam algorithm to minimize the cross entropy loss function;

[0194] The cross entropy loss function is used to update the parameters of the second attention classification model.

[0195] It should be noted that since the training set is divided into multiple training subsets, a second attention classification model can be obtained after training with each training subset.

[0196] Step 405: Select the model with the best effect on the preprocessed validation set as the trained second attention classification model, that is, the first attention classification model.

[0197] Step 406: Perform distortion correction on each VR expanded image in the test set.

[0198] Figure 8A flow chart of the method for correcting distortion of a test set provided in the embodiment of the present application is shown in FIG. Figure 8 As shown, the method for correcting distortion of a test set provided by the application embodiment of the present application includes:

[0199] Step 801: Define the coordinates of the center position of the target VR expansion image as (c x ,c y ), the coordinates of any pixel point in the target VR expansion map are (x, y), and the position coordinates of the corresponding pixel point in the corrected target VR expansion map are

[0200] Step 802: Calculate the distance r between any pixel point and the center position in the target VR expansion image.

[0201] Step 803: Calculate the distortion correction coefficient λ, λ = k1r + k2r 2 +k3r 3 +k4r 4 ; Where k1 represents the linear scaling factor, k2 is the barrel distortion factor, k3 is the radial distortion factor, which is usually 0, and k4 is the pincushion distortion factor, k1 = 1 - k2 - k3 - k4;

[0202] Step 804: Calculate the position coordinates of the corresponding pixel point in the corrected target VR expansion image:

[0203]

[0204] Step 805: Map each pixel point in the target VR expansion image to a corresponding position in the corrected target VR expansion image to obtain the corrected target VR expansion image.

[0205] Step 407: Perform sliding window cropping on each VR expanded image in the test set, cropping each VR expanded image into multiple square sub-images, and compressing the square sub-images, including:

[0206] The rectified target VR image is cropped using a square sliding window. The sliding step of the sliding window is the first sliding step, and the sliding direction of the sliding window is along the length direction of the rectified target VR image. The size of the rectified target VR image is (w, h), the size of the sliding window is (w, w), and the first sliding step is l1.

[0207] If after one or more cuttings, the remaining length of the corrected target VR unfolded image is less than the first sliding step, the sliding window uses the remaining length as the sliding step to perform the last cutting on the corrected target VR unfolded image, obtaining multiple square sub-images; denote the remaining length as l2, where l2 < l1, then the number of the obtained square sub-images is

[0208] Compress each square sub-image to the size of (m, m).

[0209] Step 408: For each VR unfolded image, input its compressed square sub-images of the size of (m, m) into the first attention classification model, and output classification results. If there are suspicious classifications in the classification results, it is determined that the classification result of the target VR unfolded image corresponding to the square sub-image belonging to the suspicious classification is a suspicious image, and it can also be determined that the target VR video is a suspicious video, and the suspicious area is marked on the corresponding VR unfolded image.

[0210] Step 409: Perform manual review on the suspicious area.

[0211] Compared with the prior art, a virtual reality VR image recognition method provided by an application embodiment of the present application has the following advantages:

[0212] A virtual reality VR image recognition method provided by an application embodiment of the present application is directed to the distortion characteristics of VR unfolded images. Through the distortion correction process in step 406, the corresponding relationship between pixel points of the target VR unfolded image and its corrected target VR unfolded image is obtained, and the distortion correction process is performed on the VR unfolded image to restore the original scene of the VR image and eliminate the negative effects brought by the distorted VR unfolded image to the first attention classification model.

[0213] A virtual reality VR image recognition method provided by an application embodiment of the present application is directed to the problem of accuracy loss of VR unfolded images. That is, for a VR unfolded image with a rectangular size, only by filling it into a square image and compressing the size and then inputting it into the first attention classification model for recognition, it will cause the VR unfolded image to lose many effective features and introduce many redundant features. Through the sliding window cutting method in step 407, the VR unfolded image is cut into multiple square sub-images using a square sliding window and respectively sent into the first attention classification model for recognition, and the recognition of the VR unfolded image is completed by combining the classification results output by multiple images, which can not only avoid the influence caused by single recognition errors but also reduce the disadvantages brought by the loss of many effective features after high-resolution images are extremely compressed.

[0214] To address the problem of detail recognition in VR unfolded images, that is, the information richness in VR unfolded images is too high, if the recognition model takes all features into consideration evenly, the classification and recognition results may be erroneous due to the neglect of subtle key information. The present application provides a virtual reality (VR) image recognition method, which introduces an attention mechanism by constructing a second attention classification model in step 403, so that the weight of effective features is greater, the effect of invalid features is reduced, and a weighted feature map is output as the final feature distribution map, thereby improving the accuracy and reliability of VR image recognition.

[0215] In order to implement the virtual reality VR image recognition method described in the embodiment of the present application, the embodiment of the present application also provides a virtual reality VR image recognition device 900, such as Figure 9 As shown, Figure 9 This is a schematic diagram of the structure of a virtual reality (VR) image recognition device 900 provided in an embodiment of the present application. The virtual reality (VR) image recognition device 900 includes:

[0216] The target acquisition unit 901 is used to acquire one or more target VR expansion images;

[0217] A cropping unit 902 is configured to crop each target VR expansion image into a plurality of square sub-images;

[0218] The model identification unit 903 is used to process each square sub-image using the first attention classification model to obtain a classification result for each square sub-image, where the classification result is a suspicious classification or a non-suspicious classification;

[0219] The determination unit 904 is configured to determine whether the target VR expansion image is a suspicious image.

[0220] In one embodiment, the virtual reality (VR) image recognition device 900 further includes:

[0221] The marking unit is used to mark the suspicious area at the corresponding position of the target VR expansion image, and the suspicious area refers to the suspicious area of ​​the square sub-image corresponding to the suspicious classification in the classification result.

[0222] It should be understood by those skilled in the art that Figure 9 The functions implemented by each unit in the VR image recognition device 900 shown can be understood by referring to the relevant description of the aforementioned virtual reality VR image recognition method. Figure 9 The functions of each unit in the VR image recognition device 900 shown can be implemented by a program running on a processor or by a specific logic circuit.

[0223] It should be noted that the virtual reality (VR) image recognition device 900 provided in the above embodiment, when performing VR image recognition, is illustrated only by the division of the aforementioned program units. In actual applications, the aforementioned processing can be assigned to different program units as needed, that is, the internal structure of the system can be divided into different program units to complete all or part of the aforementioned processing. In addition, the virtual reality (VR) image recognition device 900 provided in the above embodiment and the virtual reality (VR) image recognition method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0224] Based on the hardware implementation of the above program unit, and in order to implement the virtual reality VR image recognition method of the embodiment of the present application, the embodiment of the present application also provides an electronic device 1000, such as Figure 10 As shown, Figure 10 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application; the electronic device 1000 includes:

[0225] Storage 1002, for storing executable instructions;

[0226] The processor 1001 is configured to implement the virtual reality (VR) image recognition method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0227] In actual application, such as Figure 10 As shown, the various components in the electronic device 1000 are coupled together through the bus module 1003. It can be understood that the bus module 1003 is used to realize the connection and communication between these components. In addition to the data bus, the bus module 1003 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 10 Various buses are labeled as bus modules 1003.

[0228] An embodiment of the present application further provides a storage medium storing executable instructions. When the executable instructions are executed by at least one processor 1001, the virtual reality (VR) image recognition method provided in the embodiment of the present application is implemented.

[0229] In some embodiments, the storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface storage, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various devices including one or any combination of the above memories.

[0230] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0231] As an example, executable instructions may, but do not necessarily, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (for example, files storing one or more modules, subroutines, or code portions).

[0232] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0233] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0234] The technical solutions described in the embodiments of this application can be combined arbitrarily unless there is any conflict.

[0235] In the several embodiments provided in this application, it should be understood that the disclosed methods and electronic devices can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0236] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0237] In addition, all functional units in the embodiments of the present application can be integrated into a second processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0238] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A virtual reality (VR) image recognition method, characterized in that: include: Obtain one or more target VR expansion images; Crop each target VR expansion image into multiple square sub-images; Each square sub-image is processed using the first attention classification model to obtain a classification result for each square sub-image, which is either a suspicious classification or a non-suspicious classification; If there is a suspicious classification in the classification result, the target VR expanded image corresponding to the square sub-image belonging to the suspicious classification is determined to be a suspicious image; wherein the step of cropping each target VR expanded image into a plurality of square sub-images includes: performing distortion correction on each target VR expanded image to obtain a corrected target VR expanded image; The distortion correction of each target VR expansion image includes: Based on the coordinates of the center position of the target VR expansion image and the coordinates of any pixel point in the target VR expansion image, the distance between any pixel point and the center position in the target VR expansion image is obtained; Obtaining a distortion correction coefficient based on the distance between any pixel point and the center position and the distortion coefficient in the target VR expansion image, where the distortion coefficient includes at least one of the following: a pincushion distortion coefficient, a barrel distortion coefficient, a radial distortion coefficient, and a linear scaling coefficient; Based on the coordinates of the center position of the target VR expansion image and the distortion correction coefficient, the position of each pixel point in the target VR expansion image in the corrected target VR expansion image is obtained; Based on the position of each pixel point in the target VR expansion image in the corrected target VR expansion image, each pixel point in the target VR expansion image is mapped to the corresponding position of the corrected target VR expansion image to obtain the corrected target VR expansion image.

2. The method according to claim 1, characterized in that Also includes: If the target VR expanded image is determined to be a suspicious image, the suspicious area is marked at the corresponding position of the target VR expanded image and manually reviewed; The suspicious area refers to the suspicious area of ​​the square sub-graph corresponding to the suspicious classification in the classification result.

3. The method according to claim 2, characterized in that Also includes: The second attention classification model is trained to obtain the first attention classification model, where the first attention classification model refers to the trained second attention classification model.

4. The method according to claim 3, characterized in that The training of the second attention classification model comprises: Construct training and validation sets; Preprocess the training set and validation set; The preprocessed training set is input into the second attention classification model for training, and the parameters of the second attention classification model are updated using the loss function; The model with the best performance on the preprocessed validation set is selected as the second attention classification model.

5. The method according to claim 4, characterized in that The construction of the training set and the validation set includes: Based on the suspicious categories in the classification results, use the keywords corresponding to the suspicious categories to crawl multiple suspicious category images, and mark each suspicious category image with a suspicious category label; Based on the non-suspicious categories in the classification results, multiple non-suspicious category images are obtained from an open source dataset based on image classification, and each non-suspicious category image is labeled with a non-suspicious category label; A training set and a validation set are constructed based on suspicious and non-suspicious classified images.

6. The method according to claim 4, characterized in that The preprocessing of the training set and the validation set includes: Shuffle and batch the training set to obtain multiple training subsets; Fill each training subset and validation set with images into square graphs; The square graphs in each training subset and validation set are compressed to obtain a preprocessed training set and a preprocessed validation set, where the preprocessed training set contains multiple preprocessed training subsets.

7. The method according to claim 6, characterized in that The step of inputting the preprocessed training set into the second attention classification model for training, and updating the parameters of the second attention classification model using the loss function, comprises: Each preprocessed training subset is input into the second attention classification model for training, and the parameters of the second attention classification model are updated using the loss function.

8. The method according to claim 6, characterized in that The training set is shuffled and processed in batches to obtain multiple training subsets, including: Standardize the images in the training set and validation set to obtain a standardized training set and a standardized validation set; The standardized training set is shuffled and batched to obtain multiple training subsets.

9. The method according to claim 8, characterized in that The images in the training set and the validation set are normalized to obtain a normalized training set and a normalized validation set, including: The training set is standardized using the z-score standardization method to obtain a standardized training set; Based on the mean and variance of the training set, the validation set is standardized using the z-score standardization method to obtain the standardized validation set.

10. The method according to claim 6, characterized in that Before compressing the square graphs in each training subset and validation set, the method further includes: Perform data augmentation on the quad images in each training subset using at least one of the following methods: flipping, rotating, scaling, cropping, shifting, or introducing noise.

11. The method according to claim 4, characterized in that The step of inputting the preprocessed training set into the second attention classification model for training includes: The preprocessed training set is input into the second attention classification model based on the SENet model for training.

12. The method according to claim 11, characterized in that The preprocessed training set is input into the second attention classification model based on the SENet model for training, including: Use convolutional neural network models to extract output features of the preprocessed training set; Perform a global average pooling operation on the output features to obtain the feature distribution of each feature channel of the output features; The feature distribution of each feature channel is input into the first fully connected layer to obtain the feature distribution after dimensionality reduction; The feature distribution after dimensionality reduction is sent to the RELU activation function for activation to obtain the activated feature distribution; The activated feature distribution is sent to the second fully connected layer to obtain the feature distribution after dimension restoration; The feature distribution after dimension restoration is fed into the Sigmoid activation function to obtain the weight of each feature channel of the output feature; The output features are weighted using the weight of each feature channel of the output features to obtain the weighted output features; The weighted output features are input into the fully connected layer and classification layer of the SENet model, and the classification results of each image in the training set are output.

13. The method according to claim 4, characterized in that The updating of the parameters of the second attention classification model using the loss function includes: Use the Adam algorithm to minimize the cross entropy loss function; The cross entropy loss function is used to update the parameters of the second attention classification model.

14. The method according to any one of claims 1 to 13, characterized in that The obtaining of one or more target VR expansion images includes: Extract one or more VR images from the target VR video as the target VR image; Expand each target VR image to obtain one or more target VR expanded images.

15. The method according to claim 14, characterized in that The step of cutting each target VR expanded image into a plurality of square sub-images further includes: Use a sliding window to crop each rectified target VR image into multiple square sub-images; Compress each square subgraph to obtain a compressed square subgraph.

16. The method according to claim 15, characterized in that The sliding window is used to crop each corrected target VR expanded image into multiple square sub-images, including: The rectified target VR expanded image is cropped using a square sliding window, the sliding step of the sliding window is the first sliding step, and the sliding direction of the sliding window is along the length direction of the rectified target VR expanded image; If the remaining length of the corrected target VR expanded image is less than the first sliding step after one or more cropping operations, the sliding window performs a final cropping operation on the corrected target VR expanded image using the remaining length as the sliding step to obtain multiple square sub-images.

17. A virtual reality (VR) image recognition device, characterized in that: include: A target acquisition unit, configured to acquire one or more target VR expansion images; a cropping unit, configured to crop each target VR expanded image into a plurality of square sub-images; wherein the cropping of each target VR expanded image into a plurality of square sub-images comprises: performing distortion correction on each target VR expanded image to obtain a corrected target VR expanded image; The distortion correction of each target VR expansion image includes: Based on the coordinates of the center position of the target VR expansion image and the coordinates of any pixel point in the target VR expansion image, the distance between any pixel point and the center position in the target VR expansion image is obtained; Obtaining a distortion correction coefficient based on the distance between any pixel point and the center position and the distortion coefficient in the target VR expansion image, where the distortion coefficient includes at least one of the following: a pincushion distortion coefficient, a barrel distortion coefficient, a radial distortion coefficient, and a linear scaling coefficient; Based on the coordinates of the center position of the target VR expansion image and the distortion correction coefficient, the position of each pixel point in the target VR expansion image in the corrected target VR expansion image is obtained; Based on the position of each pixel point in the target VR expansion image in the corrected target VR expansion image, each pixel point in the target VR expansion image is mapped to the corresponding position of the corrected target VR expansion image to obtain the corrected target VR expansion image; a model recognition unit, configured to process each square sub-image using the first attention classification model to obtain a classification result for each square sub-image, wherein the classification result is a suspicious classification or a non-suspicious classification; The determination unit is used to determine whether the target VR expansion image is a suspicious image.

18. The device according to claim 17, characterized in that Also includes: The marking unit is used to mark the suspicious area at the corresponding position of the target VR expansion image, and the suspicious area refers to the suspicious area of ​​the square sub-image corresponding to the suspicious classification in the classification result.

19. An electronic device, characterized in that: include: a memory for storing executable instructions; The processor is configured to implement the virtual reality (VR) image recognition method according to any one of claims 1 to 16 when executing the executable instructions stored in the memory.

20. A storage medium, characterized in that The storage medium stores executable instructions, and when the executable instructions are executed by at least one processor, the virtual reality (VR) image recognition method according to any one of claims 1 to 16 is implemented.

Images