Image processing method, device, electronic device and storage medium
By combining the category recognition model and the LSTM model, using visual and temporal information to identify the picture collection categories, the problem of insufficient accuracy in album category recognition in the prior art is solved, and more efficient picture management and classification are achieved.
Patent Information
- Application Number
- CN202010917004.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-03
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-09-03
AI Technical Summary
The prior art relies on artificially designed rules in album category recognition, resulting in poor identification accuracy and the inability to effectively manage and classify large amounts of personal image data.
By obtaining the image set to be processed, the pre-trained category recognition model and long-term short-term memory model are used, and the categories of the image set are determined by combining the visual information and temporal information of the picture. The specific steps include inputting the picture into the category recognition model to obtain the first category probability vector, extracting the feature vector and inputting the LSTM model to obtain the second category probability vector, and fusing the two to determine the category of the picture set.
It improves the accuracy of image collection category recognition, can more accurately identify the event types and album categories of the image collection, and improves users' efficiency when viewing pictures.
Smart Images

Figure CN112069338B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of electronic devices, and more specifically, to a picture method, device, electronic device and storage medium. Background Art
[0002] Electronic devices, such as mobile phones and tablet computers, have become one of the most commonly used consumer electronic products in people's daily lives. With the development of science and technology, mobile terminals are usually equipped with cameras to achieve the shooting function, so that people can use electronic devices to shoot more and more conveniently to record the beautiful moments in life, and the captured pictures can be stored in albums for users to view. In addition, electronic devices can also generate sets of images of the same category and organize them together to form albums, but for the formed albums, users cannot accurately understand the corresponding types of these albums, such as the types of events corresponding to the albums, which affects the efficiency of users viewing pictures. Summary of the invention
[0003] In view of the above problems, the present application proposes a picture method, device, electronic device and storage medium.
[0004] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, and the method includes: obtaining a set of images to be processed; inputting each image in the set of images into a pre-trained category recognition model to obtain a first category probability vector corresponding to each image; obtaining a feature vector of each image in the set of images; inputting the feature vector of each image into a pre-trained long short-term memory model in sequence according to a preset order to obtain a second category probability vector corresponding to the set of images; and determining the category to which the set of images belongs based on the first category probability vector and the second category probability vector.
[0005] In a second aspect, an embodiment of the present application provides an image processing device, comprising: a picture set acquisition module, a first vector acquisition module, a second vector acquisition module, a third vector acquisition module and a category determination module, wherein the picture set acquisition module is used to acquire a picture set to be processed; the first vector acquisition module is used to input each picture in the picture set into a pre-trained category recognition model to obtain a first category probability vector corresponding to each picture; the second vector acquisition module is used to acquire a feature vector of each picture in the picture set; the third vector acquisition module is used to input the feature vector of each picture into a pre-trained long short-term memory model in sequence according to a preset order to obtain a second category probability vector corresponding to the picture set; the category determination module is used to determine the category to which the picture set belongs based on the first category probability vector and the second category probability vector.
[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the image processing method provided in the first aspect above.
[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the image processing method provided in the first aspect above.
[0008] The solution provided by the present application obtains a set of pictures to be processed, inputs each picture in the picture set into a pre-trained category recognition model, obtains a first category probability vector corresponding to each picture, and obtains a feature vector of each picture, inputs the feature vector corresponding to each picture into a long-term memory model in a preset order, obtains a second category probability vector corresponding to the picture set, and then determines the category to which the picture set belongs based on the first category probability vector and the second category probability vector, thereby determining the first category probability vector of each picture based on the visual information of the picture, and determining the second category probability vector of the picture set based on the time information of the picture in the picture set, and then determining the category to which the picture set belongs based on the first category probability vector and the second category probability vector, so that when determining the category of the picture set, the visual information of the picture and the time information of the picture are referred to, thereby improving the accuracy of category recognition of the picture set. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0010] Figure 1 A flow chart of a method for processing an image according to an embodiment of the present application is shown.
[0011] Figure 2 A flow chart of a picture processing method according to another embodiment of the present application is shown.
[0012] Figure 3 A flow chart of a picture processing method according to another embodiment of the present application is shown.
[0013] Figure 4 A block diagram of a picture processing device according to an embodiment of the present application is shown.
[0014] Figure 5 It is a block diagram of an electronic device according to an embodiment of the present application for executing the image processing method according to an embodiment of the present application.
[0015] Figure 6 It is a storage unit of an embodiment of the present application for storing or carrying program codes for implementing the image processing method according to an embodiment of the present application. DETAILED DESCRIPTION
[0016] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0017] At present, the penetration rate of electronic devices in daily life has almost covered all people. Among them, the camera module has become the main function of smart terminals. Users can take pictures through the camera function of electronic devices to record the moments of life, study and work. Electronic devices can save the pictures taken by people, so that users can view the pictures taken.
[0018] As the performance of electronic devices improves, a large amount of personal picture data is stored in existing electronic devices. Most of this data are photos related to the user. How to effectively manage this picture data has begun to attract the attention of various companies targeting personal consumers, resulting in the emergence of a large number of album management software using new technologies.
[0019] These album management software help users to manage their album collections intelligently and effectively from different perspectives. Among them, a very important basic function is the identification of album categories, such as album event identification. This function is based on the relevant model, analyzes the given picture group of the front-end algorithm, and automatically identifies the event type of the given picture group. The identification result can be used for subsequent recommendation scenarios. For example, the front-end algorithm outputs a group of pictures related to events such as dinners, weddings, or travel. The album event recognition algorithm analyzes the entire picture group at the same time and identifies the event type of the group of pictures, so that users can know the category of the picture group when viewing the picture group. However, most of the current solutions for album category identification are completed using artificially designed rules, and the artificially designed rules may not be considered comprehensively, resulting in poor accuracy in the identification of album categories.
[0020] In response to the above problems, the inventors have proposed the picture processing method, device, electronic device and storage medium provided in the embodiments of the present application, which refer to the visual information and time information of the pictures when determining the category of the picture set, thereby improving the accuracy of the category recognition of the picture set. The specific picture processing method will be described in detail in the subsequent embodiments.
[0021] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for processing an image provided by an embodiment of the present application. In a specific embodiment, the method for processing an image is applied to: Figure 4 The picture processing device 400 and the electronic device 100 ( Figure 5 ). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device used in this embodiment can be a smart phone, a tablet computer, a smart watch, a laptop computer, etc., which is not limited here. Figure 1 The process shown is described in detail, and the image processing method may specifically include the following steps:
[0022] Step S110: Obtain a set of pictures to be processed.
[0023] In an embodiment of the present application, an electronic device can obtain a picture set to be processed to determine the category to which the picture set to be processed belongs. The picture in the picture set to be processed may be one or more pictures. If the picture set to be processed includes one picture, it only needs to be identified by the category recognition model in the embodiment of the present application to determine the category to which the picture belongs as the category to which the picture set belongs. The embodiment of the present application is mainly explained by taking the example that the picture set to be processed includes multiple pictures, wherein the specific number of pictures in the picture set is not limited in the embodiment of the present application.
[0024] In some embodiments, the picture set can be an electronic device detecting a user's operation, and when determining the user's selection operation on a set including multiple pictures based on the detected operation, the picture set selected by the user is used as the above picture set. Among the picture sets for users to select, each picture set can include pictures belonging to a category, so that the picture set for selection can be used to identify the category of the picture set. For example, the electronic device can cluster the pictures according to the picture clustering algorithm to obtain picture sets of different categories, and these picture sets can be selected by the user; for another example, the electronic device can pre-classify pictures of the same category into the same set according to the user's operation, and generate a photo album, and the generated photo album can be selected by the user to determine the category to which the photo album belongs.
[0025] Of course, the specific method of obtaining the picture set to be processed may not be limited. It is understandable that, since the category to which the picture set belongs needs to be determined, the pictures in the picture set should belong to the same category, so that the category to which the picture set belongs to which is subsequently identified can be used as the category to which each picture in the picture set belongs.
[0026] Optionally, in an embodiment of the present application, the category to which the picture set belongs may be an event category, and images of an event category belong to images in an event, for example, a "wedding" event, a "birthday" event, a "graduation ceremony" event, a "company annual meeting" event, etc.
[0027] Step S120: input each picture in the picture set into a pre-trained category recognition model to obtain a first category probability vector corresponding to each picture.
[0028] In an embodiment of the present application, after the electronic device obtains the picture set to be processed, it can input each picture in the picture set into a pre-trained category recognition model to obtain a first category probability vector corresponding to each picture. Among them, the category recognition model is pre-trained so that it can output the probabilities of different categories according to the input pictures, and these probabilities of different categories are output in the form of probability vectors. For example, if the category recognition model can identify the probability that the picture belongs to category A, category B, category C, category D and category E, then in the probability vector (x, y, z, j, k) output by it, x corresponds to the probability of category A, y corresponds to the probability of category B, z corresponds to the probability of category C, j corresponds to the probability of category D, and k corresponds to the probability of category E.
[0029] In some embodiments, when training the category recognition model, pictures marked with category labels can be used as training samples, each training sample is marked with one or more category labels, and multiple training samples are respectively marked with corresponding category labels. The training samples are input into the initial neural network model, and the initial neural network model outputs the probabilities of different category labels. If the probabilities of different category labels output by the neural network model are different from the true probabilities corresponding to the category labels marked by the training samples themselves, the parameters of the neural network model are adjusted so that the probability output by the neural network model is close to the true probability corresponding to the category labels marked by the training samples. The above process is performed multiple times, so that the parameters of the neural network model are continuously adjusted until the difference between the probability output by the neural network model and the true probability is less than the preset difference. The neural network model is trained through multiple training samples, and the parameters of the neural network model are adjusted until the accuracy of the output result of the neural network model reaches the preset accuracy, so that a category recognition model that can be used to identify the category of the picture can be obtained. Among them, the neural network model inputs a first number of training samples, and the number of samples with accurate probabilities output is a second number, and the proportion of the second number to the first number represents the accuracy of the category recognition model.
[0030] Of course, the specific category recognition model may not be used as a limitation.
[0031] In an embodiment of the present application, after each picture in the picture set to be processed is input into a pre-trained category recognition model, the first category probability vector corresponding to each picture can be obtained. Among them, when the electronic device inputs the pictures in the picture set into the category recognition model, the order in which the pictures are input into the category recognition model can be unlimited. For example, the pictures in the picture set can be sorted according to the file size occupied by different pictures, so that pictures of different file sizes can be evenly distributed in the sorting result. When the pictures are input into the category recognition model in sequence according to the sorting, the file sizes of the pictures that the category recognition model needs to process are uniform in the whole process, so that the processing volume in a certain period of time will not be large in the whole process, which will cause the electronic device resources to be tight.
[0032] It can be understood that when using the category recognition model to identify the event probability corresponding to each picture, the event is mainly identified based on the visual information in the picture, that is, the content, style and other features of the picture. It can also be understood that after the category recognition model extracts the image features of the picture, it outputs a probability vector based on the image features.
[0033] Step S130: Obtain a feature vector of each picture in the picture set.
[0034] In an embodiment of the present application, the electronic device may also obtain a feature vector of each image in the image set to be processed, so as to subsequently determine the category probability corresponding to the entire image set through a long short-term memory model.
[0035] In some embodiments, the electronic device can extract the feature vector of each image through a pre-trained feature extraction model. The feature extraction model can be a neural network. For example, when the electronic device is a computer, neural networks such as AlexNet, VGGNet, Google Inception Net and ResNet can be selected. When the electronic device is a mobile terminal such as a mobile phone, a neural network such as Mobilenet can be selected. Among them, the entire AlexNet has 8 layers (excluding pooling layers and LRN layers) that require training parameters, the first 5 layers are convolutional layers, and the last 3 layers are fully connected layers. The last layer of AlexNet is a Softmax layer with 1000 types of outputs for classification. The LRN layer appears after the first and second convolutional layers, and the maximum pooling layer appears after two LRN layers and the last convolutional layer; VGGNet explores the relationship between the depth of the convolutional neural network and its performance. By repeatedly stacking 3*3 small convolution kernels and 2*2 maximum pooling layers, VGGNet successfully constructs a 16-19-layer deep convolutional neural network. Compared with the previous state-of-the-art network structure, the error rate of VGGNet has been greatly reduced. Since the pre-trained VGGNet in imageNet already has strong feature extraction capabilities, there is no need to retrain. The above features can be extracted based on the convolutional neural network pre-trained in imageNet. The network also further reduces the amount of calculation by decomposing the convolution. For example, a layer of 5x5 convolution can be replaced by two layers of 3x3 convolution, and a layer of 3x3 convolution can be replaced by a layer of 1x3 convolution and 3x1 convolution, greatly reducing the amount of calculation. ResNet can solve the problems of information loss and loss in the convolution layer or the fully connected layer when transmitting information. Through the residual structure, the input information is directly bypassed to the output to protect the integrity of the information. The entire network only needs to learn the difference between the input and output, simplifying the learning objectives and difficulty.
[0036] In addition, based on ResNet, SE-Net can also be used. SE-Net introduces the attention mechanism. The core idea of SENet is to learn feature weights through the network according to the loss, so that the effective feature map has a large weight, and the invalid or ineffective feature map has a small weight to train the model to achieve better results.
[0037] Of course, in the embodiment of the present application, the specific method of obtaining the feature vector may not be limited.
[0038] Step S140: inputting the feature vector of each picture into a pre-trained long short-term memory model in sequence according to a preset order to obtain a second category probability vector corresponding to the picture set.
[0039] In an embodiment of the present application, after obtaining the feature vector of each picture, the electronic device can input the feature vector of each picture into a pre-trained long short-term memory model (LSTM) in a preset order, so as to obtain the category probability vector output by the long short-term memory model, and use the category probability vector as the second category probability vector corresponding to the picture set to be processed. Since the LSTM model has a good effect on the processing of problems related to time series, and when identifying the categories corresponding to the image set, such as identifying different event categories, since the picture records of these categories obviously have a time sequence, the use of the LSTM model to identify the categories corresponding to the picture set can use the time sequence of the pictures in the picture set for identification, thereby referring to the time information of the pictures in the picture set, so that the subsequent determination of the category is more accurate.
[0040] In some embodiments, the LSTM model can be pre-trained to obtain a pre-trained LSTM model. The training of the LSTM model can refer to the aforementioned training of the category recognition model. Unlike the training of the category recognition model, the training sample of the LSTM model can be a set of images marked with category labels. After the image set is input into the LSTM model in a preset order and the output result is obtained, the output result is compared with the true result corresponding to the identified category label, and then the parameters of the LSTM model are adjusted according to the comparison result until the difference between the output result output by the LSTM model according to the picture in the input image set and the true result is less than the preset difference, thereby obtaining a pre-trained LSTM model. When the LSTM model processes the pictures input in sequence, the output corresponding to the previous picture input and the picture currently input are used as inputs, and input into the LSTM model, and the process is repeated. After the last picture is input into the LSTM model, the result output by the LSTM model is the second category probability vector corresponding to the picture set.
[0041] Optionally, the category corresponding to each probability in the probability vector output by the LSTM model can correspond to the category corresponding to each probability in the above category recognition model, so that subsequent electronic devices can determine the category to which the picture set belongs based on the obtained first probability vector and the second probability vector.
[0042] It should be noted that the order in which the electronic device obtains the first category probability vector corresponding to each picture and obtains the second category probability vector corresponding to the picture set is not limited, that is, step S120 may be before step S130 and step S140, and step S120 may also be after step S130 and step S140.
[0043] Step S150: Determine the category to which the picture set belongs based on the first category probability vector and the second category probability vector.
[0044] In an embodiment of the present application, after obtaining the first category probability vector corresponding to each image and the second category probability vector corresponding to the image set, the electronic device can determine the category to which the image set belongs based on the first category probability vector corresponding to each image and the second category probability vector. The electronic device can fuse the first category probability vector corresponding to each image and the probability information in the second category probability vector to obtain the probability that the image set belongs to different categories.
[0045] In some embodiments, the probability value at each position in the probability vector output by the category recognition model and the LSTM model corresponds to the same category. For example, each first probability vector output by the category recognition model is (x, y), and the second probability vector output by the LSTM model is (z, i), then the category corresponding to x is the same as the category corresponding to z, and the category corresponding to y is the same as the category corresponding to i.
[0046] As an implementation method, the electronic device can merge the first category probability vector corresponding to each picture with the second category probability vector respectively according to the probability values at the same position to obtain multiple fifth category probability vectors; then obtain the average value of the probability values at the same position in the multiple fifth category probability vectors to obtain a sixth category probability vector composed of multiple average values; based on the category corresponding to the maximum probability in the sixth category probability vector, determine the category to which the picture set belongs.
[0047] Among them, the first probability vectors corresponding to all pictures in the picture set can constitute a vector matrix, in which each row corresponds to the first probability vector corresponding to a picture. The data of each row in the matrix is dot-multiplied or added with the probability value at the same position (i.e., the same element) in the second category probability vector, that is, each first category probability vector is fused with the second category probability vector for each category probability, and the probability of each category after fusion is obtained. The probability of multiple categories after fusion corresponding to each picture can form the first probability vector. Then, the probability values at the same position (same element) in multiple fifth probability vectors are averaged to obtain the average value at each position, and the average value at multiple positions can form a sixth category probability vector, that is, according to the fifth category probability vector formed by the probabilities of multiple categories after fusion corresponding to multiple pictures, the average value of the probability of each category is obtained, and the average value of the obtained multiple category probabilities can constitute the sixth category probability vector. Therefore, the sixth category probability vector contains the probabilities of each category, and the category corresponding to the maximum probability is determined according to the sixth category probability vector, and the obtained category can be used as the category to which the picture set to be processed belongs.
[0048] As another implementation, the electronic device can calculate the average probability of each category for the first category probability vector corresponding to each picture in the picture set to be processed, that is, by calculating the average value for the data at the same position in each first category probability vector, the first category probability vectors of multiple pictures can be fused into one category probability vector; then the category probability vector is fused with the second category probability vector, for example, by performing dot multiplication, addition, and averaging of the data at the same position in the category probability vector, to obtain a final fused probability vector; finally, the category corresponding to the maximum probability is determined according to the final probability vector, and the obtained category can be used as the category to which the picture set to be processed belongs.
[0049] In the above implementation, in the process of fusing the category probability vectors, the categories corresponding to each position in the category probability vector do not change, that is, the positions of the fused data at each position in the fused probability vector are the same as the positions in the probability vector before fusion, thereby making it easy for the electronic device to distinguish the categories corresponding to each probability value.
[0050] Of course, the specific method of determining the category to which the picture set belongs based on the first category probability vector and the second category probability vector is not limited.
[0051] The picture processing method provided in the embodiment of the present application obtains a picture set to be processed, inputs each picture in the picture set into a pre-trained category recognition model, obtains a first category probability vector corresponding to each picture, obtains a feature vector of each picture, inputs the feature vector corresponding to each picture into a long-term memory model in a preset order, obtains a second category probability vector corresponding to the picture set, and then determines the category to which the picture set belongs based on the first category probability vector and the second category probability vector, thereby determining the first category probability vector of each picture based on the visual information of the picture, and determining the second category probability vector of the picture set based on the time information of the picture in the picture set, and then determining the category to which the picture set belongs based on the first category probability vector and the second category probability vector. Since the first category probability vector corresponding to each picture is determined based on the visual information of the picture, and the second category probability vector corresponding to the picture set is identified by referring to the time sequence information of the pictures in the picture set and identified by the LSTM model, the category to which the picture set belongs is determined based on the first category probability vector and the second category probability vector, and the visual information of the picture and the time sequence information of the pictures in the picture set can be simultaneously referred to, so that the identification of the category to which the picture set belongs is more accurate.
[0052] See also Figure 2 , Figure 2A schematic diagram of a process flow of an image processing method provided by another embodiment of the present application is shown. The image processing method is applied to the above electronic device. Figure 2 The process shown is described in detail, and the image processing method may specifically include the following steps:
[0053] Step S210: Obtain a set of pictures to be processed.
[0054] In the embodiment of the present application, step S210 can refer to the contents of the aforementioned embodiment and will not be described again here.
[0055] Step S220: input each picture in the picture set into the first network and the second network in the category recognition model.
[0056] In an embodiment of the present application, the category recognition model is a twin network model, which includes a first network and a second network. The first network and the second network have the same structure and parameters, and the first network and the second network share weights. The first network and the second network can be neural networks. The twin difference network can be trained based on a large number of pictures marked with category labels. Since the twin network model requires paired input, in actual application, each picture is input into the first network and the second network at the same time, so that data enhancement can be achieved, thereby improving the robustness of the category recognition model.
[0057] In some embodiments, the twin network model can be trained in the following manner: input the first sample image into the first network to obtain a third category probability vector, and input the second sample image into the second network to obtain a fourth category probability vector; determine the value of the total loss function of the twin network model based on the third category probability vector, the fourth category probability vector, the first true probability vector corresponding to the first sample image, and the second category probability vector of the second sample image; iteratively train the twin network model based on the value of the total loss function to obtain the trained twin network model as the category recognition model.
[0058] Among them, it can be understood that during training, two sample pictures are used as a set of data and are input into the first network and the second network of the twin network model respectively, wherein each picture is input into one network. Then, according to the output result of the first network and the true result of the input picture, the output result of the second network and the true result of the input picture, the value of the total loss function can be determined, and then according to the value of the calculated loss function, the parameters used for the two networks in the twin network model are adjusted, and the network parameters are continuously updated iteratively according to the loss value and the gradient descent method, and when the iteration termination condition is finally met, the trained twin network model is obtained. In the obtained twin network model, since the loss value continues to decrease during the iteration process, the twin network model obtained when the preset iteration termination condition is finally met can accurately identify the category probability vector corresponding to the input picture. Among them, the gradient descent method is an optimization algorithm, also commonly referred to as the steepest descent method, which is often used in machine learning and artificial intelligence to recursively approximate the minimum deviation model.
[0059] The iteration termination conditions may include: the value of the loss function is minimum, the value of the loss function is less than a preset value, or the number of iterations reaches a preset number, etc.
[0060] In some embodiments, determining a value of a total loss function of the twin network model according to the third category probability vector, the fourth category probability vector, a first true probability vector corresponding to the first sample image, and a second category probability vector of the second sample image includes:
[0061] Determining a value of a first loss function according to a difference between the third category probability vector and the fourth category probability vector;
[0062] Determining a value of a second loss function according to a difference between the first true probability vector and the second category probability vector;
[0063] According to the value of the first loss function and the value of the second loss function, the value of the total loss function of the twin network model is determined.
[0064] Specifically, the total loss function is calculated according to the following formula:
[0065]
[0066] Among them, D g =G(I i )-G(I j ), D P =P(I i )-P(I j ), P(I i) is the first network according to image I i The output probability vector, G(I i ) is image I i The corresponding true probability vector, P(I j ) is the second network according to image I j The output probability vector, G(I j ) is image I i The corresponding true probability vector. And, m s is the predefined boundary difference between the two input images, m d The predefined boundary difference between the two input images, that is, if the difference between the true probability vectors corresponding to the two images is less than m s , indicating that the two images are similar; if the difference between the true probability vectors corresponding to the two images is in m s With m d If the difference between the true probability vectors corresponding to the two pictures is greater than m, it means that the two pictures are neither the same nor different; if the difference between the true probability vectors corresponding to the two pictures is greater than m d , indicating that the two images are different. In this way, referring to the difference between the two input images, the value of the total loss function of the twin network model can be determined.
[0067] In some embodiments, a pre-trained twin network model can be stored locally on an electronic device. When the electronic device needs to obtain the first category probability vector corresponding to a picture in a picture set, the twin network model can be called locally, thereby improving the efficiency of obtaining the first category probability vector corresponding to the picture; the pre-trained twin network model can be stored on a server, and the electronic device can obtain the first category probability vector returned from the server by sending the picture to the server. In this way, when the twin network model is large and the electronic device is a mobile terminal, the storage space of the electronic device can be effectively saved, as well as the operating resources of the electronic device.
[0068] Of course, in the image processing method provided in the embodiment of the present application, other models (such as the model for extracting feature vectors in subsequent steps, the LSTM model, etc.) can also be stored locally in the electronic device or in the server in the above manner.
[0069] Step S230: Obtain the category probability vector output by the first network or the category probability vector output by the second network as the first category probability vector.
[0070] In an embodiment of the present application, when each picture is input into the category recognition model, the picture is simultaneously input into the first network and the second network in the above twin network model, and the structure, parameters, and weights of the first network and the second network in the twin network model are the same, therefore, the output results of the first network and the second network for the picture should be the same, and the category probability vector output by any network can be taken as the first category probability vector corresponding to the picture.
[0071] Step S240: Obtain the time information of each picture.
[0072] In an embodiment of the present application, the electronic device may also determine the preset order before inputting each picture into the LSTM model in the preset order. Since the pictures are taken in a time sequence when recording different types of picture sets, especially when recording pictures of different events, the time information of each picture may be obtained, based on which the time sequence of the pictures may be determined.
[0073] In some implementations, the electronic device may obtain the shooting time of each picture, and determine the shooting time of each picture as the time information of each picture. It is understandable that the electronic device will record the shooting time when shooting a picture, so the time information of the picture can be determined based on this, and the time sequence of the pictures in the picture set can be determined.
[0074] Step S250: Determine the preset order according to the sequence of the time information of each picture.
[0075] In the embodiment of the present application, after determining the time information of each picture, the preset order of the pictures when input into the LSTM model can be determined according to the sequence of the time information of each picture. Among them, the electronic device can sort the pictures in the picture set according to the time information of each picture in the order from first to last, so as to obtain the preset order.
[0076] Step S260: inputting the feature vector of each picture into a pre-trained long short-term memory model in sequence according to a preset order to obtain a second category probability vector corresponding to the picture set.
[0077] Step S270: Determine the category to which the picture set belongs based on the first category probability vector and the second category probability vector.
[0078] In the embodiment of the present application, step S260 and step S270 can refer to the contents of the aforementioned embodiment and will not be described again here.
[0079] The image processing method provided in the embodiment of the present application determines the first category probability vector corresponding to each image based on the visual information of the image, and the second category probability vector corresponding to the image set refers to the temporal information of the images in the image set and is identified through the LSTM model. Therefore, the category to which the image set belongs is determined based on the first category probability vector and the second category probability vector. The visual information of the image and the temporal information of the images in the image set can be referred to at the same time, so that the identification of the category to which the image set belongs is more accurate. In addition, the twin network model is used to obtain the first category probability vector corresponding to each image. Since the twin network model requires paired input, that is, each image is input into the first network and the second network at the same time, it can actually have a data enhancement effect, improve the robustness of the category recognition model, and then improve the accuracy of the first category probability vector corresponding to each obtained image.
[0080] See also Figure 3 , Figure 3 A schematic diagram of a process flow of a picture processing method provided by another embodiment of the present application is shown. The picture processing method is applied to the above electronic device. Figure 3 The process shown is described in detail, and the image processing method may specifically include the following steps:
[0081] Step S310: In response to a selection operation on a target theme album, pictures in the target theme album are obtained.
[0082] In an embodiment of the present application, a plurality of theme albums may be pre-stored in the electronic device. These theme albums are albums pre-formed by the electronic device according to a method for generating theme albums, or are albums formed after the user divides pictures of the same category.
[0083] In some embodiments, the electronic device may pre-cluster designated pictures to obtain clustering results, and then generate at least one theme album based on pictures belonging to the same category in the clustering results, wherein the at least one theme album includes the target theme album.
[0084] The designated images are clustered, so that images belonging to the same category after clustering can be obtained, and the images belonging to the same category are clustered as a category, so as to obtain a clustering result of at least one category. Specifically, when clustering, the images can be clustered according to the features of the images of different categories, such as the environmental information when the images were taken. The clustering algorithm can be the HDBSCAN algorithm, the K-Means algorithm, the MeanShift algorithm, etc., which are not limited here.
[0085] Step S320: When the number of pictures in the target theme album is greater than a specified number, importance scores of different pictures are generated according to the historical operation data of different pictures in the target theme album.
[0086] In an embodiment of the present application, when it is necessary to determine the category to which the target theme album belongs, if the number of pictures in the target theme album is too large, the subsequent processing volume will be large. Therefore, before executing the subsequent category determination process, the number of pictures in the target theme album can also be determined, and then the number of pictures can be compared with the specified number; according to the comparison result, when it is determined that the number of pictures is greater than the specified number, the specified number of pictures can be screened out as a picture set to be processed to determine the category to which the target theme album belongs; according to the comparison result, when it is determined that the number of pictures is less than or equal to the specified number, the pictures in the target theme album can be directly used as a picture set to be processed. Among them, the specific value of the specified number may not be used as a limitation, for example, it can be 5, or it can be 10, etc.
[0087] In some embodiments, the electronic device can generate importance scores for different pictures based on the historical operation data of different pictures in the target theme album. The historical operation data may include: the number of times a picture is marked as liked, the number of times a picture is shared, the number of times a picture is edited, and the viewing time of the picture. Each picture can be marked as liked by the user, and the electronic device can record the number of times each picture is marked as liked by the user; the electronic device can also respond to the user's operation to share the picture, such as sharing it to other devices, and for example, sharing it to a social platform, etc., and the electronic device can record the number of times each picture is shared; the electronic device can also respond to the editing operation of the picture and edit the picture, such as beautifying it, etc., and the electronic device can record the number of times the picture is edited; the electronic device can also record the user's viewing time of the picture when the user views the picture. It is understandable that historical operation data such as the number of times a picture is marked as liked, the number of times a picture is shared, the number of times a picture is edited, and the viewing time of a picture can all reflect the user's importance and preference for the picture, and therefore can reflect the importance of the picture. Subsequently, the importance score of the picture can be generated based on these historical operation data, and then after sorting the pictures according to the importance score of the picture, it is possible to sort the pictures according to their importance. Of course, the specific historical operation records are not limited to the above content. For example, the historical operation data can also include the number of times a picture is viewed, etc. It is understandable that the historical operation data of a picture can reflect the user's importance to the picture, that is, it can reflect the importance of the picture.
[0088] When an electronic device generates an importance score of an image based on the historical operation data of the image, if the historical operation record includes multiple records, each record can be quantified and normalized; then, a weighted sum is performed based on the weight of each record and the normalized data corresponding to each record to obtain the importance score of the image.
[0089] Step S330: According to the importance score, obtain the specified number of pictures whose importance scores meet the preset score condition from the pictures in the target theme album, and use the picture set consisting of the specified number of pictures as the picture set to be processed.
[0090] In an embodiment of the present application, after obtaining the importance scores of pictures in the target theme album, the electronic device can filter a specified number of pictures from the pictures in the target theme album as an image set to be processed based on the importance scores of the pictures, thereby reducing the amount of processing in the subsequent processing process.
[0091] In some embodiments, the electronic device can sort the pictures in the picture set according to the importance scores of the pictures, in descending order of importance scores, and then obtain the first specified number of pictures from the sorting results according to the sorting results as the picture set to be processed. Of course, the preset score condition can also be other, for example, the importance score is greater than the specified score, etc.
[0092] Step S340: Obtain a set of pictures to be processed.
[0093] Step S350: input each picture in the picture set into a pre-trained category recognition model to obtain a first category probability vector corresponding to each picture.
[0094] Step S360: Obtain a feature vector of each picture in the picture set.
[0095] Step S370: input the feature vector of each picture into the pre-trained long short-term memory model in sequence according to a preset order to obtain a second category probability vector corresponding to the picture set.
[0096] Step S380: Determine the category to which the picture set belongs based on the first category probability vector and the second category probability vector.
[0097] In the embodiment of the present application, steps S340 to S380 can refer to the contents of the aforementioned embodiment and will not be repeated here.
[0098] In the embodiment of the present application, the category to which the picture set belongs determined by the electronic device can be used as the category of the target theme album, such as "wedding" or "birthday". After determining the category of the target theme album, the electronic device can add a category label to the target theme album, so that the user can know the category when viewing the album.
[0099] In some embodiments, after determining the category to which the atlas to be processed belongs, the electronic device can also select the picture with the highest relevance to the category from the atlas as the cover picture according to the determined category; then generate the cover of the atlas according to the cover picture, so as to further facilitate the user to know the category of the atlas when viewing the atlas. Among them, when the electronic device calculates the relevance between the pictures in the atlas and the category to which the atlas belongs, it can be determined according to the number of content features of the category contained in the pictures. The more content features of the category are contained, the higher the relevance. For example, corresponding to the wedding category, the pictures of the wedding category will contain content features such as "white wedding dress", "bride", "hall", "food", and "church". Based on this, the number of these content features contained in the pictures in the atlas of the wedding category can be determined, and then the determined proportion can be used as the relevance according to the proportion of the determined number to the total number of content features.
[0100] In the above implementation, when there are multiple pictures with the highest relevance, the picture with the highest importance score can be selected from the multiple pictures according to the importance score of the picture as the cover picture. The importance score can refer to the above content. In the above manner, a picture that is more relevant to a category can be determined from the picture collection as the cover picture, so that the user can know the category of the picture collection when viewing the picture collection.
[0101] The picture processing method provided in the embodiment of the present application determines the number of pictures of the selected target subject album according to the user's selection operation on the subject album. When the number of pictures is greater than the specified number, the specified number of pictures are screened out as the picture set to be processed by calculating the importance score, thereby reducing the amount of calculation of the electronic device. When determining the category to which the picture set belongs, since the first category probability vector corresponding to each picture is determined based on the visual information of the picture, and the second category probability vector corresponding to the picture set is identified by referring to the time sequence information of the pictures in the picture set through the LSTM model, the category to which the picture set belongs is determined based on the first category probability vector and the second category probability vector, and the visual information of the picture and the time sequence information of the pictures in the picture set can be referred to at the same time, so that the identification of the category to which the picture set belongs is more accurate.
[0102] See also Figure 4, which shows a structural block diagram of a picture processing device 400 provided in an embodiment of the present application. The picture processing device 400 applies the above-mentioned electronic device, and the picture processing device 400 includes: a picture set acquisition module 410, a first vector acquisition module 420, a second vector acquisition module 430, a third vector acquisition module 440 and a category determination module 450. Among them, the picture set acquisition module 410 is used to acquire a picture set to be processed; the first vector acquisition module 420 is used to input each picture in the picture set into a pre-trained category recognition model to obtain a first category probability vector corresponding to each picture; the second vector acquisition module 430 is used to obtain a feature vector of each picture in the picture set; the third vector acquisition module 440 is used to input the feature vector of each picture into a pre-trained long short-term memory model in a preset order in sequence to obtain a second category probability vector corresponding to the picture set; the category determination module 450 is used to determine the category to which the picture set belongs based on the first category probability vector and the second category probability vector.
[0103] In some embodiments, the image processing device 400 may further include: a time information acquisition module and an order determination module. The time information acquisition module is used to acquire the time information of each image before inputting the feature vector of each image into the pre-trained long short-term memory model in a preset order to obtain the second category probability vector corresponding to the image set; the order determination module is used to determine the preset order according to the order of the time information of each image.
[0104] In this implementation, the time information acquisition module can be specifically used to: acquire the shooting time of each picture; and determine the shooting time of each picture as the time information of each picture.
[0105] In some embodiments, the category recognition model is a twin network model, and the twin network model includes a first network and a second network. The image processing device 400 may also include: a fourth vector acquisition module, a loss acquisition module, and an iterative training module. The fourth vector acquisition module is used to input the first sample image into the first network to obtain a third category probability vector, and input the second sample image into the second network to obtain a fourth category probability vector; the loss acquisition module is used to determine the value of the total loss function of the twin network model based on the third category probability vector, the fourth category probability vector, the first true probability vector corresponding to the first sample image, and the second category probability vector of the second sample image; the iterative training module is used to iteratively train the twin network model according to the value of the total loss function to obtain the trained twin network model as the category recognition model.
[0106] In this embodiment, the loss acquisition module can be specifically used to: determine the value of the first loss function based on the difference between the third category probability vector and the fourth category probability vector; determine the value of the second loss function based on the difference between the first true probability vector and the second category probability vector; determine the value of the total loss function of the twin network model based on the value of the first loss function and the value of the second loss function.
[0107] In this embodiment, the first vector acquisition module 420 can be specifically used to: input each picture in the picture set into the first network and the second network; obtain the category probability vector output by the first network or the category probability vector output by the second network as the first category probability vector.
[0108] In some embodiments, the category determination module 450 can be specifically used to: merge the first category probability vector corresponding to each image with the second category probability vector respectively according to the probability values at the same position to obtain multiple third category probability vectors; obtain the average value of the probability values at the same position in the multiple third category probability vectors to obtain a fourth category probability vector composed of multiple average values; determine the category to which the image set belongs based on the category corresponding to the maximum probability in the fourth category probability vector.
[0109] In some embodiments, the picture set acquisition module 410 can be specifically used to: respond to a selection operation on a target theme album, and acquire pictures in the target theme album; when the number of pictures in the target theme album is greater than a specified number, generate importance scores of different pictures based on historical operation data of different pictures in the target theme album; based on the importance scores, acquire the specified number of pictures whose importance scores meet preset score conditions from the pictures in the target theme album, and use the picture set consisting of the specified number of pictures as the picture set to be processed.
[0110] In some embodiments, the image processing device 400 may further include a clustering module and an album generation module. The clustering module is used to cluster the specified images to obtain clustering results before the image in the target theme album is obtained in response to the selection operation for the target theme album; the album generation module is used to generate at least one theme album according to the images belonging to the same category in the clustering results, and the at least one theme album includes the target theme album.
[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.
[0112] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.
[0113] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.
[0114] In summary, the solution provided by the present application obtains a set of pictures to be processed, inputs each picture in the picture set into a pre-trained category recognition model, obtains a first category probability vector corresponding to each picture, and obtains a feature vector of each picture, inputs the feature vector corresponding to each picture into a long-term memory model in a preset order, obtains a second category probability vector corresponding to the picture set, and then determines the category to which the picture set belongs based on the first category probability vector and the second category probability vector, thereby determining the first category probability vector of each picture based on the visual information of the picture, and determining the second category probability vector of the picture set based on the time information of the picture in the picture set, and then determining the category to which the picture set belongs based on the first category probability vector and the second category probability vector, so that when determining the category of the picture set, the visual information of the picture and the time information of the picture are referred to, thereby improving the accuracy of category recognition of the picture set.
[0115] Please refer to Figure 5 , which shows a structural block diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, a smart watch, a laptop computer, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.
[0116] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.
[0117] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.
[0118] Please refer to Figure 6 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.
[0119] The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.
[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for processing an image, characterized in that: Applied to electronic equipment, the method comprises: Get the image set to be processed; Input each picture in the picture set into a pre-trained category recognition model to obtain a first category probability vector corresponding to each picture, and construct a plurality of vector matrices corresponding to the first category probability vectors, wherein each row in the vector matrix corresponds to the first category probability vector corresponding to a picture, and the category recognition model is used to recognize the category according to the visual information of the picture; Obtaining a feature vector for each picture in the picture set; Inputting the feature vector of each picture into a pre-trained long short-term memory model in sequence according to a preset order to obtain a second category probability vector corresponding to the picture set; Performing point multiplication or addition of the data of each row in the vector matrix and the probability value at the same position in the second category probability vector to obtain the fused probability of each category, and the fused probabilities of multiple categories corresponding to each picture constitute a fifth category probability vector, to obtain multiple fifth category probability vectors; Obtain an average value of probability values at the same position in the plurality of fifth category probability vectors, to obtain a sixth category probability vector consisting of the plurality of average values; The category to which the picture set belongs is determined based on the category corresponding to the maximum probability in the sixth category probability vector.
2. The method according to claim 1, characterized in that Before inputting the feature vector of each picture into the pre-trained long short-term memory model in a preset order to obtain the second category probability vector corresponding to the picture set, the method further includes: Obtaining time information of each picture; The preset order is determined according to the sequence of the time information of each picture.
3. The method according to claim 2, characterized in that The obtaining of the time information of each picture includes: Obtaining the shooting time of each picture; The shooting time of each picture is determined as the time information of each picture.
4. The method according to claim 1, characterized in that: The category recognition model is a twin network model, and the twin network model includes a first network and a second network. Before inputting each picture in the picture set into the pre-trained category recognition model to obtain the first category probability vector corresponding to each picture, the method further includes: Inputting the first sample image into the first network to obtain a third category probability vector, and inputting the second sample image into the second network to obtain a fourth category probability vector; Determine a value of a total loss function of the twin network model according to the third category probability vector, the fourth category probability vector, a first true probability vector corresponding to the first sample image, and a second category probability vector of the second sample image; According to the value of the total loss function, the twin network model is iteratively trained to obtain the trained twin network model as the category recognition model.
5. The method according to claim 4, characterized in that The determining a value of a total loss function of the twin network model according to the third category probability vector, the fourth category probability vector, a first true probability vector corresponding to the first sample image, and a second category probability vector of the second sample image includes: Determining a value of a first loss function according to a difference between the third category probability vector and the fourth category probability vector; Determining a value of a second loss function according to a difference between the first true probability vector and the second category probability vector; According to the value of the first loss function and the value of the second loss function, the value of the total loss function of the twin network model is determined.
6. The method according to claim 4, characterized in that The step of inputting each picture in the picture set into a pre-trained category recognition model to obtain a first category probability vector corresponding to each picture includes: Input each picture in the picture set into the first network and the second network; Obtain a category probability vector output by the first network or a category probability vector output by the second network as the first category probability vector.
7. The method according to claim 1, characterized in that The step of obtaining a set of pictures to be processed includes: In response to a selection operation on a target theme album, obtaining pictures in the target theme album; When the number of pictures in the target theme album is greater than a specified number, generating importance scores of different pictures according to historical operation data of different pictures in the target theme album; According to the importance score, the specified number of pictures whose importance scores meet the preset score condition are obtained from the pictures in the target theme album, and the picture set consisting of the specified number of pictures is used as the picture set to be processed.
8. The method according to claim 7, characterized in that Before the step of responding to the selection operation on the target theme album and obtaining the pictures in the target theme album, the method further includes: Cluster the specified pictures and obtain the clustering results; At least one theme album is generated according to the pictures belonging to the same category in the clustering results, and the at least one theme album includes the target theme album.
9. A picture processing device, characterized in that: The device comprises: a picture set acquisition module, a first vector acquisition module, a second vector acquisition module, a third vector acquisition module and a category determination module, wherein: The picture set acquisition module is used to acquire the picture set to be processed; The first vector acquisition module is used to input each picture in the picture set into a pre-trained category recognition model to obtain a first category probability vector corresponding to each picture, and construct a vector matrix corresponding to a plurality of the first category probability vectors, wherein each row in the vector matrix corresponds to the first category probability vector corresponding to a picture, and the category recognition model is used to identify the category according to the visual information of the picture; The second vector acquisition module is used to acquire a feature vector of each picture in the picture set; The third vector acquisition module is used to input the feature vector of each picture into the pre-trained long short-term memory model in sequence according to a preset order to obtain the second category probability vector corresponding to the picture set; The category determination module is used to perform point multiplication or addition of the data of each row in the vector matrix with the probability value at the same position in the second category probability vector to obtain the probability of each fused category, and the probabilities of multiple fused categories corresponding to each image constitute a fifth category probability vector to obtain multiple fifth category probability vectors; obtain the average value of the probability values at the same position in the multiple fifth category probability vectors to obtain a sixth category probability vector composed of multiple average values; determine the category to which the picture set belongs based on the category corresponding to the maximum probability in the sixth category probability vector.
10. An electronic device, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-8.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 8.
