Intelligent recognition and classification method of cloud photo frame photos based on machine learning

Abstract

The invention relates to the technical field of intelligent photo classification, in particular to a cloud photo frame photo intelligent recognition and classification method based on machine learning; the invention performs preliminary classification based on photo image files and basic metadata thereof, so that a user can quickly locate photos according to simple retrieval of time, place and camera model; generates secondary retrieval classification groups based on analysis and calculation of RGB color values, HSV space and grayscale values, performs feature extraction according to image parameters such as color, hue and brightness of each photo, can accurately reflect image information contained in the photo, and is suitable for photo classification tasks with high color accuracy requirements; the invention constructs a convolutional neural network to abstractly extract scene features and quantify them through output values, thereby improving the accuracy and adaptability of the cloud photo frame photo intelligent recognition and classification method, and enhancing the scalability and adaptability of the intelligent recognition and classification method to different usage scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent photo classification, and in particular to a cloud photo frame intelligent recognition and classification method based on machine learning. Background Art

[0002] Cloud photo frame is a digital photo frame that uses cloud computing technology to network terminal services and realizes cloud services through cloud servers. It combines the advantages of traditional digital photo frames and cloud computing technology. Cloud photo frames can not only display photos, but also upload, store, share and manage photos instantly through cloud servers. Users only need to upload photos to the cloud, and the cloud photo frame can automatically synchronize and display them, making photo sharing and management more convenient.

[0003] However, with the increasing frequency of network interactions, cloud photo frames are facing many image processing-related problems such as image compression, encoding, decoding, storage, and transmission. People are gradually entering an era of reading pictures, and the number of image-related applications is increasing, which puts higher requirements on the classification and search capabilities of album applications. In the actual photo retrieval process, people cannot intuitively analyze the specific place names and location information in the photos; at the same time, people's estimates of the specific shooting time of the photos are often relative. With the increase in the date of use, the number of photos is increasing day by day. It is becoming more and more difficult to find specific photos, and the classification requirements are becoming higher and higher. The limitations of traditional photo classification methods are also becoming increasingly obvious.

[0004] In addition, the existing cloud photo frame classification methods are often limited to basic data such as time, location, and photo source, and lack specific feature quantification of photo content. The retrieval of a specific photo is often based on people's subjective judgment of the image content in the thumbnail, which reduces the user experience of the cloud photo frame.

[0005] Therefore, it is necessary to propose a cloud photo frame photo intelligent recognition and classification method based on machine learning to solve the above problems. Summary of the invention

[0006] The purpose of the present invention is to solve the problems existing in the background technology and propose a cloud photo frame photo intelligent recognition and classification method based on machine learning;

[0007] The purpose of the present invention can be achieved by the following technical solution: a cloud photo frame photo intelligent recognition and classification method based on machine learning, specifically comprising the following steps:

[0008] Step 1: Extract image metadata and construct first-level label classification groups;

[0009] The cloud photo frame obtains all photo image files and their basic metadata from the cloud server. The basic metadata includes shooting date (Date Time Original), shooting location latitude value (GPS Latitude), shooting location longitude value (GPS Longitude), image width (Image Width), image height (image Height), camera model (Model), exposure time (Exposure Time), ISO sensitivity value (ISO Speed ​​Ratings) and shutter speed (Shutter speed Value). All photo image files are classified and grouped according to the values ​​of each basic metadata. The specific classification principles are as follows:

[0010] Classify by shooting date: classify the photo image files with the same shooting date into the same group, and obtain several shooting date groups;

[0011] Classification by shooting location: photo image files with the same shooting location latitude and longitude are determined to have the same shooting location and classified into the same group, thereby obtaining a number of shooting location groups;

[0012] Classification by shooting size: photo image files with the same image width and image height are judged to have the same shooting size and classified into the same group, thereby obtaining a number of shooting size groups;

[0013] Classify by camera model: classify the photo image files with the same camera model into the same group, and obtain several camera model groups;

[0014] Classification by camera parameters: classify photo image files with the same exposure time into the same group, and obtain a number of exposure time groups; classify photo image files with the same ISO sensitivity value into the same group, and obtain a number of ISO sensitivity value groups; classify photo image files with the same shutter speed into the same group, and obtain a number of shutter speed groups;

[0015] The shooting date grouping, shooting location grouping, shooting size grouping, camera model grouping, exposure time grouping, photosensitivity grouping and shutter speed grouping obtained after classification and grouping are collectively referred to as the primary label classification group. Each basic metadata is recorded as a primary label.

[0016] Step 2: RGB three-channel decomposition, space conversion and grayscale value extraction;

[0017] Each photo image file is numbered, and the number symbol is i, i=1, 2, 3, ..., n; wherein n is the total number of photo image files.

[0018] Obtain the pixel resolution Xi×Yi of each photo image file i, and locate all pixels in the photo image file with the positioning coordinates (x, y), where x and y are the sorting order of the pixels along the long and short sides of the image file, respectively, where 0≤x≤Xi; 0≤y≤Yi.

[0019] The color of each pixel is decomposed into three channels of R, G and B to obtain the three-channel color value of each pixel (x, y), including the red color value R(x, y), the green color value G(x, y) and the blue color value B(x, y); the specific value ranges of R(x, y), G(x, y) and B(x, y) are all 0 to 255.

[0020] By formula

[0021]

[0022] The color of each pixel (x, y) is converted from the three channels of R, G and B to the HSV space, where θ(x, y) is the hue difference, H(x, y) is the hue, S(x, y) is the saturation, and V(x, y) is the lightness.

[0023] The grayscale value r(x, y) of each pixel is calculated by the formula r(x, y)=int{K1×R(x, y)+K2×G(x, y)+K3×B(x, y)}, where int is the rounding calculation symbol; K1, K2 and K3 are preset weight factors and K1+K2+K3=1; r(x, y) is any integer between 0 and 255, representing the grayscale value corresponding to the pixel with the positioning coordinates (x, y).

[0024] Step 3: Extract pixel features and construct secondary label classification groups;

[0025] Pixel features of all photo image files are extracted from three dimensions: RGB color value, HSV space and grayscale value. Specifically:

[0026] Feature extraction of RGB color value dimension:

[0027] For all pixels (x, y) in a specific photo image file, the formula Calculate the recognition value n of each pixel (x, y) on the RGB three channels for a certain red color value R0∈[0,255] R (x, y), the identification value n for a certain green color value G0∈[0,255] G (x, y) and the identification value n for a certain blue color value B0∈[0,255] B(x, y), where 0≤x≤Xi, 0≤y≤Yi; where R0, G0 and B0 are respectively the preset red color value, green color value and blue color value for pixel color value identification.

[0028] As a preferred embodiment of the present invention, for the entire photo image file i, the formula Calculate the pixel proportion hi(R) of the red color value R∈[0,255], the pixel proportion hi(G) of the green color value G∈[0,255], and the pixel proportion hi(B) of the blue color value B∈[0,255], where int is the rounding symbol.

[0029] By formula Calculate the red eigenvalue Ei(R), green eigenvalue Ei(G) and blue eigenvalue Ei(B) of the photo image file i. k(R), k(G) and k(B) are three preset weight factors, where 0≤R≤255; 0≤G≤255; 0≤B≤255.

[0030] Feature extraction in HSV space dimension:

[0031] By formula Calculate the recognition value n of each pixel (x, y) for a certain combination of hue H, saturation S and lightness V HSV (x, y).

[0032] As a preferred embodiment of the present invention, for the entire photo image file i, the formula Calculate the HSV space eigenvalue Ei(HSV) of photo image file i.

[0033] Feature extraction of gray value dimension:

[0034] By formula Calculate the recognition value n of each pixel (x, y) for a certain gray value r r (x, y).

[0035] As a preferred embodiment of the present invention, for the entire photo image file i, the formula Calculate the grayscale eigenvalue Ei(r) of the photo image file i, where k(r) is a set of preset weight factors, where 0≤r≤255.

[0036] The pixel feature extraction process is traversed through the photo image file number i=1, 2, 3, ..., n to obtain the red feature value Ei(R), green feature value Ei(G), blue feature value Ei(B), HSV space feature value Ei(HSV) and gray feature value Ei(r) of each photo image file i.

[0037] As a preferred embodiment of the present invention, the red eigenvalue Ei(R), the green eigenvalue Ei(G), the blue eigenvalue Ei(B), the HSV space eigenvalue Ei(HSV) and the grayscale eigenvalue Ei(r) are collectively referred to as secondary labels. Photo image files with the same secondary label are classified into a group to obtain a secondary classification group, including a red eigenvalue group, a green eigenvalue group, a blue eigenvalue group, an HSV space eigenvalue group and a grayscale eigenvalue group.

[0038] Step 4: Extract scene features and construct three-level label classification groups;

[0039] Define the machine learning process based on convolutional neural networks, including the following steps:

[0040] S401, classification label preset;

[0041] A set of three-level labels are preset, including:

[0042] Scene labels: residential interior E01, store E02, factory E03, high-rise building E04, street E05, traffic road E06, beach E07 and wilderness and mountain area E08;

[0043] Seasonal environment labels: precipitation E09, snow E10, and clear E11;

[0044] Shooting element labels: People E12, Vehicles E13, Animals E14 and Natural Scenery E15.

[0045] S402 establishes a convolutional neural network;

[0046] A 6-layer convolutional neural network model was established, including the input layer, the first convolutional layer, the second convolutional layer, the third convolutional layer, the first fully connected layer and the second fully connected layer.

[0047] The input layer contains the compressor and preprocessor;

[0048] The compressor converts the image height and width to 640×360 by compressing and cropping the original image.

[0049] The preprocessor of the first convolutional layer filters the input image by compression, cropping and Gaussian transformation. The Gaussian transformation formula is: Where σ is the Gaussian standard deviation, * is the convolution operator, μ(x, y) is the Gaussian convolution kernel; f(x, y) is the original input image, and y0(x, y) is the output image of the input layer.

[0050] The first, second, and third convolutional layers all contain convolutional, activation, and pooling layers.

[0051] The first convolutional layer contains n1 nodes, and the output formula of the first convolutional layer is: Where j1 = 1, 2, 3, ..., n1; is the weight factor of the first convolutional layer, is the bias factor of the first convolutional layer; where F is the activation function of the activation layer.

[0052] The second convolutional layer contains n2 nodes, and the output formula of the second convolutional layer is: Where j2 = 1, 2, 3, ..., n2; is the weight factor of the second convolutional layer, is the bias factor of the second convolutional layer; where F is the activation function of the activation layer.

[0053] The third convolutional layer contains n3 nodes, and the output formula of the third convolutional layer is: Where j3 = 1, 2, 3, ..., n3; is the weight factor of the third convolutional layer, is the bias factor of the second convolutional layer; where F is the activation function of the activation layer.

[0054] The activation function F of the activation layer in the first, second and third convolutional layers all adopts the LeakyReLU function, and its function expression is: Where z is the input value of the LeakyReLU function.

[0055] The pooling layers in the first, second and third convolutional layers perform noise reduction and redundant data removal through average pooling operations, where the average pooling is to average the pixel output results in the pooling area.

[0056] The first fully connected layer contains n4 nodes, and its output formula is: Where j4 = 1, 2, 3, ..., n4; is the weight factor of the first fully connected layer, is the bias factor of the first fully connected layer; where F is the activation function of the activation layer.

[0057] The second fully connected layer contains 25 nodes, and its output formula is Where p = 01, 02, 03, ..., 15; where w p is the weight factor of the second fully connected layer, b p is the bias factor of the second fully connected layer;

[0058] S403 training image collection;

[0059] Collect several photos containing three-level label elements, specifically, 50 photos of residential interior, store, factory, high-rise, street, traffic road, beach, wilderness and mountain, precipitation, snow, clear, people, transportation, animals and natural scenery, totaling 750 photos. Each photo is numbered, and the number symbol is q, q = 1, 2, 3, ..., 750.

[0060] As a preferred embodiment of the present invention, the 750 photos are marked with three-level labels. For photos containing a certain three-level label element, the three-level label Ep corresponding to the photo is set to 1; otherwise, Ep is set to 0. Where p = 01, 02, 03, ..., 15.

[0061] S404 Neural Network Training and Machine Learning;

[0062] The 750 photos marked with three-level labels are input as original images into the 6-layer convolutional neural network model, and the output value corresponding to each original image is The specific value of is equal to the third-level label Ep with the same number symbol p.

[0063] Perform training operations on the convolutional neural network and record all weight factors and bias factors obtained after training.

[0064] Test the convolutional neural network: repeat step S403 to obtain 750 new photos as original images and input them into the 6-layer convolutional neural network after training; then record the obtained output values ​​Fp=F01, F02, F03, ..., F15.

[0065] By formula The test success rate is calculated. If the test success rate S is greater than the preset threshold, the training is determined to be successful, all weight factors and bias factors obtained through training are output, and the process proceeds to step S405.

[0066] Otherwise, the training is determined to have failed, and step S404 is re-executed until the condition is met: the test success rate S is greater than a preset threshold.

[0067] S405 model application;

[0068] All photo image files i are input as original images into the 6-layer neural network to obtain a set of output values ​​F01, F02, F03, ..., F15. The output values ​​greater than the preset threshold are extracted and the image metadata i is marked with the corresponding three-level labels.

[0069] The photo image files with the same third-level labels are classified into a group to obtain three-level classification groups, including residential interior group, store group, factory group, high-rise group, street group, traffic road group, beach group, wilderness and mountain group, precipitation group, snow group, sunny group, person group, vehicle group, animal group and natural scenery group.

[0070] Step 5: Multi-label intelligent classification and display;

[0071] The photo image files are finally classified and stored according to the complete set of photo features, namely, primary tags, secondary tags and tertiary tags; a photo storage folder is created for each type of primary tag, secondary tag and tertiary tag. Each photo image file is accurately copied to a storage folder that matches its tag name, so that the name of each photo storage folder is consistent with the common tag name of the photo image files contained therein.

[0072] In the display interface of the cloud photo frame, photos are displayed according to different tag categories based on the user's selection or preset display rules; when the user enters multiple classification tags, duplicate photos contained in the photo storage folder corresponding to each input classification tag are retrieved.

[0073] Step 6: Image search and multi-label classification verification;

[0074] For each photo image file i, an L-bit binary code containing label classification result information is assigned to each image. Each label corresponds to an independent binary bit in the binary code, and when the binary bit is 1, it indicates that the label exists, and when the binary bit is 0, it indicates that the label does not exist.

[0075] All binary bits in the binary code are numbered, the numbering symbol is U, and U = 1, 2, 3, ..., L.

[0076] Establish a mechanism for image search and multi-label classification result verification, specifically:

[0077] Image search mechanism: For each photo image file i, obtain the binary code containing the label classification result information, and use the Euclidean distance formula: Calculate the feature similarity Euclidean Distance (i1, i2) of any photo image files i1 and i2, where i1∈i and i2∈i; i1≠i2. Where fu(i1) is the u-th binary bit in the binary code of the photo image file i1; fu(i2) is the u-th binary bit in the binary code of the photo image file i2.

[0078] Whenever a user opens a photo image file i1, the feature similarity between i1 and all other photo image files is calculated and numerically compared. Then, the five photo image files with the highest feature similarity to the image file i1 are extracted and displayed.

[0079] Furthermore, the user is asked through the UI interface which of the five displayed photo image files is most similar to the image file i1.

[0080] Verification mechanism of multi-label classification results based on image search results:

[0081] When the user selects a photo that is closest to the user, the binary code of the selected photo is obtained and the loss function is used to calculate the binary code. Calculate the loss factor LOSS.

[0082] When the loss coefficient LOSS is greater than a preset threshold, it is determined that the multi-label classification result for the photo image file i1 is poor.

[0083] Every preset detection period T, count the number of times r the user opens the photo image file and the loss coefficient LOSS generated each time, and use the formula Calculate the failure coefficient Distort. When the failure coefficient is greater than the preset threshold, the multi-label classification result verification result is determined to be False, and the multi-label classification is unreliable. The user is prompted and the process returns to step 4 to re-execute the convolutional neural network training.

[0084] Compared with the prior art, the present invention has the following beneficial effects:

[0085] (1) The present invention constructs a convolutional neural network to abstractly extract scene features and quantify them through output values. According to the quantization results, the preset three-level labels are matched. The convolutional neural network can automatically learn and extract effective image features from the original pixels without manually designing feature extraction algorithms. This feature improves the accuracy and adaptability of the cloud photo frame photo intelligent recognition and classification method; and by adjusting the preset three-level labels and adjusting parameters, and retraining the convolutional neural network, the classification rules can be adjusted, which improves the scalability of the intelligent recognition and classification method and its adaptability to different usage scenarios;

[0086] (2) The present invention performs preliminary classification based on photo image files and their basic metadata, and generates a primary label classification group, so that users can quickly locate photos based on simple searches for time, location and camera model. Its classification process and rules have good compatibility. Based on the analysis and calculation of RGB color values, HSV space and grayscale values, the present invention generates a secondary search classification group, and extracts features according to image parameters such as color, hue, and brightness of each photo, which can accurately reflect the image information contained in the photo. It is suitable for photo classification tasks that require high color accuracy.

[0087] (3) The present invention adds rules for classification according to camera parameters such as exposure time, ISO sensitivity value, and shutter speed in the label classification group, providing a new retrieval perspective for cloud photo frame users. When photos under specific parameters are needed, they can quickly locate the corresponding classification, improve search efficiency, and select the best quality photos as preferred photos; for photographers, they can clearly see the performance effects of photos under different parameters, so as to have a deeper understanding of the impact of parameters such as exposure time, ISO sensitivity value and shutter speed on photo quality, helping photographers analyze whether their parameter settings under different shooting environments are reasonable. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0089] Figure 1 is a flow chart of the method of the present invention;

[0090] Figure 2 It is a schematic diagram of the HSV space of the present invention;

[0091] Figure 3 This is a topological diagram of the 6-layer convolutional neural network structure of the present invention. DETAILED DESCRIPTION

[0092] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0093] See also Figure 1 As shown, the cloud photo frame photo intelligent recognition and classification method based on machine learning includes the following steps:

[0094] Step 1: Extract image metadata and construct first-level label classification groups;

[0095] The cloud photo frame obtains all photo image files and their basic metadata from the cloud server. The basic metadata includes shooting date (Date Time Original), shooting location latitude value (GPS Latitude), shooting location longitude value (GPS Longitude), image width (Image Width), image height (image Height), camera model (Model), exposure time (Exposure Time), ISO sensitivity value (ISO Speed ​​Ratings) and shutter speed (Shutter speed Value). All photo image files are classified and grouped according to the values ​​of each basic metadata. The specific classification principles are as follows:

[0096] Classify by shooting date: classify the photo image files with the same shooting date into the same group, and obtain several shooting date groups;

[0097] Classification by shooting location: photo image files with the same shooting location latitude and longitude are determined to have the same shooting location and classified into the same group, thereby obtaining a number of shooting location groups;

[0098] Classification by shooting size: photo image files with the same image width and image height are judged to have the same shooting size and classified into the same group, thereby obtaining a number of shooting size groups;

[0099] Classify by camera model: classify the photo image files with the same camera model into the same group, and obtain several camera model groups;

[0100] Classification by camera parameters: classify photo image files with the same exposure time into the same group, and obtain a number of exposure time groups; classify photo image files with the same ISO sensitivity value into the same group, and obtain a number of ISO sensitivity value groups; classify photo image files with the same shutter speed into the same group, and obtain a number of shutter speed groups;

[0101] The shooting date grouping, shooting location grouping, shooting size grouping, camera model grouping, exposure time grouping, photosensitivity grouping and shutter speed grouping obtained after classification and grouping are collectively referred to as the primary label classification group. Each basic metadata is recorded as a primary label.

[0102] Step 2: RGB three-channel decomposition, space conversion and grayscale value extraction;

[0103] Each photo image file is numbered, and the number symbol is i, i=1, 2, 3, ..., n; wherein n is the total number of photo image files.

[0104] Obtain the pixel resolution Xi×Yi of each photo image file i, and locate all pixels in the photo image file with the positioning coordinates (x, y), where x and y are the sorting order of the pixels along the long and short sides of the image file, respectively, where 0≤x≤Xi; 0≤y≤Yi.

[0105] The color of each pixel is decomposed into three channels of R, G and B to obtain the three-channel color value of each pixel (x, y), including the red color value R(x, y), the green color value G(x, y) and the blue color value B(x, y); the specific value ranges of R(x, y), G(x, y) and B(x, y) are all 0 to 255.

[0106] By formula

[0107]

[0108] The color of each pixel (x, y) is converted from the three channels of R, G and B to the HSV space, where θ(x, y) is the hue difference, H(x, y) is the hue, S(x, y) is the saturation, and V(x, y) is the lightness.

[0109] See also Figure 2 As shown, the HSV space is a cone color space model, which includes all colors in the color domain of the photo image file.

[0110] The grayscale value r(x, y) of each pixel is calculated by the formula r(x, y)=int{K1×R(x, y)+K2×G(x, y)+K3×B(x, y)}, where int is the rounding calculation symbol; K1, K2 and K3 are preset weight factors and K1+K2+K3=1; r(x, y) is any integer between 0 and 255, representing the grayscale value corresponding to the pixel with the positioning coordinates (x, y).

[0111] Step 3: Extract pixel features and construct secondary label classification groups;

[0112] Pixel features of all photo image files are extracted from three dimensions: RGB color value, HSV space and grayscale value. Specifically:

[0113] Feature extraction of RGB color value dimension:

[0114] For all pixels (x, y) in a specific photo image file, the formula Calculate the recognition value n of each pixel (x, y) on the RGB three channels for a certain red color value R0∈[0,255] R (x, y), the identification value n for a certain green color value G0∈[0,255] G(x, y) and the identification value n for a certain blue color value B0∈[0,255] B (x, y), where 0≤x≤Xi, 0≤y≤Yi; where R0, G0 and B0 are respectively the preset red color value, green color value and blue color value for pixel color value identification.

[0115] Furthermore, for the entire photo image file i, the formula Calculate the pixel proportion hi(R) of the red color value R∈[0,255], the pixel proportion hi(G) of the green color value G∈[0,255], and the pixel proportion hi(B) of the blue color value B∈[0,255], where int is the rounding symbol.

[0116] By formula Calculate the red eigenvalue Ei(R), green eigenvalue Ei(G) and blue eigenvalue Ei(B) of the photo image file i. k(R), k(G) and k(B) are three preset weight factors, where 0≤R≤255; 0≤G≤255; 0≤B≤255.

[0117] Feature extraction in HSV space dimension:

[0118] By formula Calculate the recognition value n of each pixel (x, y) for a certain combination of hue H, saturation S and lightness V HSV (x, y).

[0119] Furthermore, for the entire photo image file i, the formula Calculate the HSV space eigenvalue Ei(HSV) of photo image file i.

[0120] Feature extraction of gray value dimension:

[0121] By formula Calculate the recognition value n of each pixel (x, y) for a certain gray value r r (x, y).

[0122] Furthermore, for the entire photo image file i, the formula Calculate the grayscale eigenvalue Ei(r) of the photo image file i, where k(r) is a set of preset weight factors, where 0≤r≤255.

[0123] The pixel feature extraction process is traversed through the photo image file number i=1, 2, 3, ..., n to obtain the red feature value Ei(R), green feature value Ei(G), blue feature value Ei(B), HSV space feature value Ei(HSV) and gray feature value Ei(r) of each photo image file i.

[0124] Furthermore, the red eigenvalue Ei(R), the green eigenvalue Ei(G), the blue eigenvalue Ei(B), the HSV spatial eigenvalue Ei(HSV) and the grayscale eigenvalue Ei(r) are collectively referred to as secondary labels. The photo image files with the same secondary label are classified into a group to obtain secondary classification groups, including a red eigenvalue group, a green eigenvalue group, a blue eigenvalue group, an HSV spatial eigenvalue group and a grayscale eigenvalue group.

[0125] Step 4: Extract scene features and construct three-level label classification groups;

[0126] Define the machine learning process based on convolutional neural networks, including the following steps:

[0127] S401, classification label preset;

[0128] A set of three-level labels are preset, including:

[0129] Scene labels: residential interior E01, store E02, factory E03, high-rise building E04, street E05, traffic road E06, beach E07 and wilderness and mountain area E08;

[0130] Seasonal environment labels: precipitation E09, snow E10, and clear E11;

[0131] Shooting element labels: People E12, Vehicles E13, Animals E14 and Natural Scenery E15.

[0132] S402 establishes a convolutional neural network;

[0133] See also Figure 3 As shown, a 6-layer convolutional neural network model is established, including an input layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a first fully connected layer, and a second fully connected layer.

[0134] The input layer contains the compressor and preprocessor;

[0135] The compressor converts the image height and width to 640×360 by compressing and cropping the original image.

[0136] The preprocessor of the first convolutional layer filters the input image by compression, cropping and Gaussian transformation. The Gaussian transformation formula is: Where σ is the Gaussian standard deviation, * is the convolution operator, μ(x, y) is the Gaussian convolution kernel; f(x, y) is the original input image, and y0(x, y) is the output image of the input layer.

[0137] The first, second, and third convolutional layers all contain convolutional, activation, and pooling layers.

[0138] The first convolutional layer contains n1 nodes, and the output formula of the first convolutional layer is: Where j1 = 1, 2, 3, ..., n1; is the weight factor of the first convolutional layer, is the bias factor of the first convolutional layer; where F is the activation function of the activation layer.

[0139] The second convolutional layer contains n2 nodes, and the output formula of the second convolutional layer is: Where j2 = 1, 2, 3, ..., n2; is the weight factor of the second convolutional layer, is the bias factor of the second convolutional layer; where F is the activation function of the activation layer.

[0140] The third convolutional layer contains n3 nodes, and the output formula of the third convolutional layer is: Where j3 = 1, 2, 3, ..., n3; is the weight factor of the third convolutional layer, is the bias factor of the second convolutional layer; where F is the activation function of the activation layer.

[0141] The activation function F of the activation layer in the first, second and third convolutional layers all adopts the LeakyReLU function, and its function expression is: Where z is the input value of the LeakyReLU function.

[0142] The pooling layers in the first, second and third convolutional layers perform noise reduction and redundant data removal through average pooling operations, where the average pooling is to average the pixel output results in the pooling area.

[0143] The first fully connected layer contains n4 nodes, and its output formula is: Where j4 = 1, 2, 3, ..., n4; is the weight factor of the first fully connected layer, is the bias factor of the first fully connected layer; where F is the activation function of the activation layer.

[0144] The second fully connected layer contains 25 nodes, and its output formula is Where p = 01, 02, 03, ..., 15; where w p is the weight factor of the second fully connected layer, b p is the bias factor of the second fully connected layer;

[0145] S403 training image collection;

[0146] Collect several photos containing three-level label elements, specifically, 50 photos of residential interior, store, factory, high-rise, street, traffic road, beach, wilderness and mountain, precipitation, snow, clear, people, transportation, animals and natural scenery, totaling 750 photos. Each photo is numbered, and the number symbol is q, q = 1, 2, 3, ..., 750.

[0147] Furthermore, the 750 photos are marked with three-level labels. For photos containing a certain three-level label element, the three-level label Ep corresponding to the photo is set to 1; otherwise, Ep is set to 0. Where p = 01, 02, 03, ..., 15.

[0148] S404 Neural Network Training and Machine Learning;

[0149] The 750 photos marked with three-level labels are input as original images into the 6-layer convolutional neural network model, and the output value corresponding to each original image is The specific value of is equal to the third-level label Ep with the same number symbol p.

[0150] Perform training operations on the convolutional neural network and record all weight factors and bias factors obtained after training.

[0151] Test the convolutional neural network: repeat step S403 to obtain 750 new photos as original images and input them into the 6-layer convolutional neural network after training; then record the obtained output values ​​Fp=F01, F02, F03, ..., F15.

[0152] By formula The test success rate is calculated. If the test success rate S is greater than the preset threshold, the training is determined to be successful, all weight factors and bias factors obtained through training are output, and the process proceeds to step S405.

[0153] Otherwise, the training is determined to have failed, and step S404 is re-executed until the condition is met: the test success rate S is greater than a preset threshold.

[0154] S405 model application;

[0155] All photo image files i are input as original images into the 6-layer neural network to obtain a set of output values ​​F01, F02, F03, ..., F15. The output values ​​greater than the preset threshold are extracted and the image metadata i is marked with the corresponding three-level labels.

[0156] The photo image files with the same third-level labels are classified into a group to obtain three-level classification groups, including residential interior group, store group, factory group, high-rise group, street group, traffic road group, beach group, wilderness and mountain group, precipitation group, snow group, sunny group, person group, vehicle group, animal group and natural scenery group.

[0157] Step 5: Multi-label intelligent classification and display;

[0158] The photo image files are finally classified and stored according to the complete set of photo features, namely the primary tags, secondary tags and tertiary tags; a photo storage folder is created for each type of primary tags, secondary tags and tertiary tags, and each photo image file is accurately copied to a storage folder that matches its tag name, so that the name of each photo storage folder is consistent with the common tag name of the photo image files contained therein.

[0159] In the display interface of the cloud photo frame, photos are displayed according to different tag categories based on the user's selection or preset display rules; when the user enters multiple classification tags, duplicate photos contained in the photo storage folder corresponding to each input classification tag are retrieved.

[0160] Step 6: Image search and multi-label classification verification;

[0161] For each photo image file i, an L-bit binary code containing label classification result information is assigned to each image. Each label corresponds to an independent binary bit in the binary code, and when the binary bit is 1, it indicates that the label exists, and when the binary bit is 0, it indicates that the label does not exist.

[0162] All binary bits in the binary code are numbered, the numbering symbol is U, and U = 1, 2, 3, ..., L.

[0163] Establish a mechanism for image search and multi-label classification result verification, specifically:

[0164] Image search mechanism: For each photo image file i, obtain the binary code containing the label classification result information, and use the Euclidean distance formula: Calculate the feature similarity Euclidean Distance (i1, i2) of any photo image files i1 and i2, where i1∈i and i2∈i; i1≠i2. Where fu(i1) is the u-th binary bit in the binary code of the photo image file i1; fu(i2) is the u-th binary bit in the binary code of the photo image file i2.

[0165] Whenever a user opens a photo image file i1, the feature similarity between i1 and all other photo image files is calculated and numerically compared. Then, the five photo image files with the highest feature similarity to the image file i1 are extracted and displayed.

[0166] Furthermore, the user is asked through the UI interface which of the five displayed photo image files is most similar to the image file i1.

[0167] Verification mechanism of multi-label classification results based on image search results:

[0168] When the user selects a photo that is closest to the user, the binary code of the selected photo is obtained and the loss function is used to calculate the binary code. Calculate the loss factor LOSS.

[0169] When the loss coefficient LOSS is greater than a preset threshold, it is determined that the multi-label classification result for the photo image file i1 is poor.

[0170] Every preset detection period T, count the number of times r the user opens the photo image file and the loss coefficient LOSS generated each time, and use the formula Calculate the failure coefficient Distort. When the failure coefficient is greater than the preset threshold, the multi-label classification result verification result is determined to be False, and the multi-label classification is unreliable. The user is prompted and the process returns to step 4 to re-execute the convolutional neural network training.

[0171] In addition, we make a simple implementation summary based on the actual scenario to better understand the behavior: for example, a user needs to find a group of photos of "photos of blue beaches in 2024". If the photos stored by the user have not been manually marked in advance (that is, there is no description of the photos), then the system first obtains all photo image files and their basic metadata from the cloud server. Obviously, the basic metadata required here is: shooting date (DateTimeOriginal) 2024, and other metadata are the same; then it is necessary to extract pixel features and construct secondary label classification groups. Obviously, the secondary label feature required here is "blue", and the specific feature extraction process will not be repeated; then it is necessary to extract the tertiary label, that is, "beach". This third-level label is obviously the most similar to the photos marked with the third-level label in the 750 photos mentioned above. The photo is marked with the third-level label. Therefore, it will be searched in the search similar to the beach, and the label "beach" can be added in subsequent operations; obviously, the third-level label of this core data is supplemented by training. In the future, as more and more training data is stored in the server, the efficiency and success rate of retrieval will become higher and higher.

[0172] It should be understood that the terms “include” and “comprising” used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0173] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and the claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and the claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations;

[0174] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A cloud photo frame photo intelligent recognition and classification method based on machine learning, characterized in that: The following steps are involved: Step 1: Extract image metadata and construct a first-level label classification group; the cloud photo frame obtains all photo image files and their basic metadata from the cloud server, and classifies and groups all photo image files according to the values ​​of each basic metadata. The groups obtained after classification and grouping are collectively referred to as first-level label classification groups; each basic metadata is recorded as a first-level label; Step 2: RGB three-channel decomposition, spatial conversion and grayscale value extraction; the red color value R (x, y), green color value G (x, y) and blue color value B (x, y) of each pixel are obtained by RGB three-channel decomposition; The hue, saturation and brightness of each pixel are obtained through spatial transformation; the grayscale value of each pixel is extracted; Step 3: Extract pixel features and construct secondary label classification groups; extract pixel features from all photo image files from three dimensions: RGB color value, HSV space and grayscale value, and classify and group them according to the extracted pixel feature values; the groups obtained after classification and grouping are collectively referred to as secondary label classification groups; each pixel feature value is recorded as a secondary label; Step 4: Extract scene features and construct three-level label classification groups; define a machine learning process based on a convolutional neural network, use the convolutional neural network to extract features, obtain output values ​​about the three-level labels and perform three-level label matching on the photo image files; classify the photo image files with the same three-level labels into a group to obtain a three-level classification group; Step 5: Multi-label intelligent classification and display; finally classify and store the photo image files according to the complete set of photo features, namely, primary tags, secondary tags and tertiary tags; create a photo storage folder for each type of primary tags, secondary tags and tertiary tags; accurately copy each photo image file to a storage folder that matches its tag name, so that the name of each photo storage folder is consistent with the common tag name of the photo image files contained therein; in the display interface of the cloud photo frame, display photos according to different tag categories according to the user's choice or preset display rules; when the user enters multiple classification tags, retrieve the duplicate photos contained in the photo storage folder corresponding to each input classification tag; Step 6: Image search and multi-label classification verification; Assign an L-bit binary code containing the label classification result to each image, number and calibrate each label; calculate the feature similarity between images through Euclidean distance; display the five images most similar to the selected image for users to choose; calculate the loss coefficient of the classification result based on the similar images selected by the user using the loss function, and judge whether the classification result is distorted based on the preset threshold; regularly count user interaction data and calculate the failure coefficient. When the failure coefficient exceeds the threshold, the multi-label classification result is judged to be unreliable and return to the training stage to re-execute the convolutional neural network training.

2. The method for intelligent recognition and classification of cloud photo frames based on machine learning according to claim 1 is characterized in that: The basic metadata includes: Shooting date, shooting location latitude, shooting location longitude, image width, image height, camera model, exposure time, ISO sensitivity value and shutter speed.

3. The method for intelligent recognition and classification of cloud photo frames based on machine learning according to claim 2 is characterized in that: The specific process of classifying and grouping all photo image files according to the values ​​of each basic metadata is as follows: Classify by shooting date: classify the photo image files with the same shooting date into the same group, and obtain several shooting date groups; Classification by shooting location: photo image files with the same shooting location latitude and longitude are determined to have the same shooting location and classified into the same group, thereby obtaining a number of shooting location groups; Classification by shooting size: photo image files with the same image width and image height are judged to have the same shooting size and classified into the same group, thereby obtaining a number of shooting size groups; Classify by camera model: classify the photo image files with the same camera model into the same group, and obtain several camera model groups; Classification by camera parameters: classify photo image files with the same exposure time into the same group, and obtain a number of exposure time groups; classify photo image files with the same ISO sensitivity value into the same group, and obtain a number of ISO sensitivity value groups; classify photo image files with the same shutter speed into the same group, and obtain a number of shutter speed groups; The shooting date grouping, shooting location grouping, shooting size grouping, camera model grouping, exposure time grouping, sensitivity value grouping and shutter speed grouping obtained after classification and grouping are collectively referred to as a first-level label classification group; each basic metadata is recorded as a first-level label.

4. The method for intelligent recognition and classification of cloud photo frames based on machine learning according to claim 1, characterized in that: The specific process of RGB three-channel decomposition is: Number each photo image file, the number symbol is i, i=1, 2, 3, ..., n; where n is the total number of photo image files; Obtain the pixel resolution Xi×Yi of each photo image file i, locate all pixels in the photo image file, and the positioning coordinates are (x, y), where x and y are the sorting order of the pixels along the long side and the short side of the image file, respectively, where 0≤x≤Xi; 0≤y≤Yi; Decompose the color of each pixel into three channels: R, G, and B, and obtain the three-channel color value of each pixel (x, y), including the red color value R(x, y), the green color value G(x, y), and the blue color value B(x, y); the specific value ranges of R(x, y), G(x, y), and B(x, y) are all 0 to 255.

5. The method for intelligent recognition and classification of cloud photo frames based on machine learning according to claim 4 is characterized in that: The specific process of space conversion is: By formula Convert the color of each pixel (x, y) from the three channels of R, G and B to the HSV space, where θ(x, y) is the hue difference, H(x, y) is the hue, S(x, y) is the saturation, and V(x, y) is the lightness.

6. The method for intelligent recognition and classification of cloud photo frames based on machine learning according to claim 5, characterized in that: The specific process of gray value extraction is: The gray value r(x, y) of each pixel is calculated by the formula r(x, y)=int{K1×R(x, y)+K2×G(x, y)+K3×B(x, y)}, where int is the rounding calculation symbol; Where K1, K2 and K3 are preset weight factors and K1+K2+K3=1; where r(x, y) is any integer between 0 and 255, representing the grayscale value corresponding to the pixel with the positioning coordinates (x, y).

7. The method for intelligent recognition and classification of cloud photo frames based on machine learning according to claim 6, characterized in that: The specific process of extracting pixel features from three dimensions: RGB color value, HSV space and grayscale value is as follows: For all pixels (x, y) in a specific photo image file, the formula Calculate the recognition value of each pixel (x, y) on the RGB three channels for a certain red color value R∈[0,255] , for a certain green color value G∈[0,255], the recognition value And the recognition value for a certain blue color value B∈[0,255] , where 0≤x≤Xi, 0≤y≤Yi; For the entire photo image file i, the formula Calculate the pixel proportion hi(R) of the red color value R∈[0,255], the pixel proportion hi(G) of the green color value G∈[0,255], and the pixel proportion hi(B) of the blue color value B∈[0,255], where int is the rounding symbol; By formula Calculate the red eigenvalue Ei(R), green eigenvalue Ei(G) and blue eigenvalue Ei(B) of the photo image file i; k(R), k(G) and k(B) are three preset weight factors; 0≤R≤255; 0≤G≤255; 0≤B≤255; By formula Calculate the recognition value of each pixel (x, y) for a certain combination of hue H, saturation S and lightness V ; For the entire photo image file i, the formula Calculate the HSV space eigenvalue Ei (HSV) of the photo image file i; By formula Calculate the recognition value of each pixel (x, y) for a certain gray value r ; For the entire photo image file i, the formula Calculate the grayscale eigenvalue Ei(r) of the photo image file i, where k(r) is a set of preset weight factors, where 0≤r≤255; and int is the rounding symbol.

8. The method for intelligent recognition and classification of cloud photo frames based on machine learning according to claim 1, characterized in that: The specific process of classifying and grouping according to the extracted pixel feature values ​​is as follows: The feature extraction process traverses the photo image file number i=1, 2, 3, ..., n; the red feature value Ei(R), green feature value Ei(G), blue feature value Ei(B), HSV space feature value Ei(HSV) and gray feature value Ei(r) of each photo image file i are obtained; The red eigenvalue Ei(R), green eigenvalue Ei(G), blue eigenvalue Ei(B), HSV space eigenvalue Ei(HSV) and grayscale eigenvalue Ei(r) are collectively referred to as secondary labels; The photo image files with the same secondary label are classified into a group to obtain secondary classification groups, including a red feature value group, a green feature value group, a blue feature value group, an HSV space feature value group and a grayscale feature value group.

9. The method for intelligent recognition and classification of cloud photo frames based on machine learning according to claim 1, characterized in that: The machine learning process specifically includes the following steps: S401, classification label preset; A set of 15 three-level labels containing various photo shooting elements and backgrounds is preset; S402 establishes a convolutional neural network; Establish a 6-layer convolutional neural network model, including the input layer, the first convolutional layer, the second convolutional layer, the third convolutional layer, the first fully connected layer, and the second fully connected layer; The input layer contains the compressor and preprocessor; The compressor converts the image height and width into 640×360 pixels by compressing and cropping the original image; The preprocessor of the first convolutional layer filters the input image by compression, cropping and Gaussian transformation. The Gaussian transformation formula is: ; where σ is the Gaussian standard deviation, * is the convolution operator, μ(x, y) is the Gaussian convolution kernel; f(x, y) is the input original image, where y0(x, y) is the output image of the input layer; The first, second, and third convolutional layers all contain convolutional, activation, and pooling layers; The first convolutional layer contains n1 nodes, and the output formula of the first convolutional layer is: ; where j1=1, 2, 3, ..., n1; where * is the convolution operator; where is the weight factor of the first convolutional layer, is the bias factor of the first convolutional layer; where F is the activation function of the activation layer; The second convolutional layer contains n2 nodes, and the output formula of the second convolutional layer is: ; where j2=1, 2, 3, ..., n2; where * is the convolution operator; where is the weight factor of the second convolutional layer, is the bias factor of the second convolutional layer; where F is the activation function of the activation layer; The third convolutional layer contains n3 nodes, and the output formula of the third convolutional layer is: ; where j3=1, 2, 3, ..., n3; where * is the convolution operator; where is the weight factor of the third convolutional layer, is the bias factor of the second convolutional layer; where F is the activation function of the activation layer; The activation function F of the activation layer in the first, second and third convolutional layers all adopts the LeakyReLU function, and its function expression is: , where z is the input value of the LeakyReLU function; The pooling layers in the first, second and third convolutional layers perform noise reduction and redundant data removal through average pooling operations, wherein the average pooling is to average the pixel output results of the pooling area; The first fully connected layer contains n4 nodes, and its output formula is: ; where j4=1, 2, 3, ..., n4; where * is the convolution operator; where is the weight factor of the first fully connected layer, is the bias factor of the first fully connected layer; where F is the activation function of the activation layer; The second fully connected layer contains 25 nodes, and its output formula is ; where p = 01, 02, 03, ..., 15; where * is the convolution operator; where is the weight factor of the second fully connected layer, is the bias factor of the second fully connected layer; S403 training image collection; Collect 50 photos of each of the 15 preset third-level label elements, for a total of 750 photos; number each photo, the number symbol is q, q = 1, 2, 3, ..., 750; The 750 photos are marked with three-level labels; for a photo containing a certain three-level label element, the three-level label Ep corresponding to the photo is set to 1; otherwise, Ep is set to 0; wherein p=01, 02, 03, ..., 15; S404 Neural Network Training and Machine Learning; The 750 photos marked with three-level labels are input into the 6-layer convolutional neural network model as original images, and the output value of the 6-layer convolutional neural network model corresponding to each original image is The specific value of is equal to the third-level label Ep with the same number symbol p; Perform training operations on the convolutional neural network and record all weight factors and bias factors obtained after training. Testing the convolutional neural network: repeating step S403 to obtain 750 new photos as original images and inputting them into the 6-layer convolutional neural network after training; then recording the obtained output values ​​Fp=F01, F02, F03, ..., F15; By formula Calculate the test success rate; if the test success rate S is greater than the preset threshold, the training is determined to be successful, all weight factors and bias factors obtained from the training are output, and the process proceeds to step S405; Otherwise, the training is determined to have failed, and step S404 is re-executed until the condition is met: the test success rate S is greater than a preset threshold; S405 model application; All photo image files i are input as original images into a 6-layer neural network to obtain a set of output values ​​F01, F02, F03, ..., F15; the output values ​​greater than the preset threshold are extracted and the image metadata i is marked with the corresponding three-level labels.

10. The method for intelligent recognition and classification of cloud photo frames based on machine learning according to claim 9, characterized in that: The preset third-level tags are actual marks and descriptions of the photo content.

Images