Image management method and device, model training method and device, electronic equipment, server and medium

Abstract

The invention discloses an image management method and device, a model training method and device, electronic equipment, a server and a medium, and belongs to the technical field of artificial intelligence. The method comprises the steps that a first image sequence is obtained, the first image sequence comprises M first images selected by a user, and M is an integer larger than or equal to 2; model reasoning is carried out based on the first image sequence and the N first to-be-selected images, target to-be-selected images are output, the target to-be-selected images are at least one to-be-selected image in the N first to-be-selected images, and N is a positive integer.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image management method, a model training method, a device, an electronic device, a server, and a medium. Background Art

[0002] As imaging technology in electronic devices improves, users are increasingly using them to capture and record their daily lives, leading to a growing number of images stored on these devices. Since images take up a significant amount of storage space on electronic devices, efficiently managing images in albums has become a pressing issue.

[0003] In the related art, when users search for photos in batches, they need to swipe through the album interface to search and select the desired images one by one. This process requires users to spend a lot of time to filter the images, making the image search process cumbersome and time-consuming. As such, the existing image management methods are inefficient, and there is an urgent need to provide efficient and convenient image management methods to meet user needs. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide an image management method, model training method, device, electronic device, server and medium, which can simplify the user's manual image screening operation steps and improve the efficiency and convenience of image management.

[0005] In a first aspect, an embodiment of the present application provides an image management method, which includes: obtaining a first image sequence, the first image sequence including M first images selected by a user, where M is an integer greater than or equal to 2; performing model inference based on the first image sequence and N first candidate images, and outputting a target candidate image, where the target candidate image is at least one candidate image among the N first candidate images, where N is a positive integer.

[0006] In second aspect, an embodiment of the present application provides a model training method, which includes: obtaining a first image sample set and a second image sample set, the first image sample set including images to be selected and M images selected by the user, the second image sample set including random images and M images selected by the user, and the random images are generated or obtained based on predetermined rules; training an image recommendation model on the first image sample set and the second image sample set to obtain model parameters of the image recommendation model.

[0007] In a third aspect, an embodiment of the present application provides an image management device, which includes: a processing module, the processing module being used to: obtain a first image sequence, the first image sequence including M first images selected by a user, where M is an integer greater than or equal to 2; perform model inference based on the first image sequence and N first candidate images, and output a target candidate image, where the target candidate image is at least one candidate image among the N first candidate images, where N is a positive integer.

[0008] In a fourth aspect, an embodiment of the present application provides a model training device, which includes: a processing module, which is used to: obtain a first image sample set and a second image sample set, wherein the first image sample set includes images to be selected and M images selected by the user, and the second image sample set includes random images and M images selected by the user, and the random images are generated or obtained based on predetermined rules; train an image recommendation model based on the first image sample set and the second image sample set to obtain model parameters of the image recommendation model.

[0009] In a fifth aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0010] In a sixth aspect, an embodiment of the present application provides a server comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the second aspect are implemented.

[0011] In a seventh aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0012] In an eighth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect or the second aspect.

[0013] In a ninth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0014] In an embodiment of the present application, an electronic device obtains a first image sequence, which includes a first candidate image and M first images selected by the user; performs model inference on the above-mentioned first image sequence and outputs a predicted probability of the first candidate image; and outputs a target candidate image based on the predicted probability of the first candidate image. In this solution, by using multiple images selected by the user as a reference, the electronic device can use model inference to quickly predict the target image that the user wants to select. Since the user only needs to select a small number of images, the electronic device can infer the user's preferences or needs based on these images and output the target candidate image. In this way, the user's manual image screening steps are simplified, improving the efficiency and convenience of image management. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of an image management method provided in some embodiments of the present application;

[0016] Figure 2 A schematic diagram of the calculation process of the self-attention mechanism provided for some embodiments of the present application;

[0017] Figure 3 A schematic diagram of the calculation process of the self-attention mechanism provided for some embodiments of the present application;

[0018] Figure 4 A schematic diagram of the model structure of an image recommendation model provided in some embodiments of the present application;

[0019] Figure 5 Schematic diagram of K, V, and Embedding cache design provided for some embodiments of this application;

[0020] Figure 6 A schematic diagram of the attention calculation process after adding cache provided in some embodiments of the present application;

[0021] Figure 7 A flowchart of a model training method provided for some embodiments of the present application;

[0022] Figure 8 A schematic diagram of the structure of an image management device provided in some embodiments of the present application;

[0023] Figure 9 A schematic diagram of the structure of a model training device provided in some embodiments of the present application;

[0024] Figure 10 A schematic structural diagram of an electronic device provided in some embodiments of the present application;

[0025] Figure 11 Schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0027] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] The terms "at least one of" and "at least one of" in the specification and claims of this application refer to any one, any two, or a combination of more than two of the objects included. For example, at least one of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c", and "a, b, and c", where a, b, and c can be single or multiple. Similarly, "M items" refers to two or more, and its meaning is similar to "at least one item".

[0029] The image management method provided in the embodiments of the present application may be executed by an image processing device, which may be an electronic device, or a functional module or functional entity in the electronic device. The image management method provided in the embodiments of the present application will be described below using an electronic device executing the image management method as an example.

[0030] The image management method provided in the embodiment of the present application can be applied to the scenario of managing images in the photo album of an electronic device. In the photo album management scenario, the user's management requirements for the photo album generally include image classification, image deletion, etc.

[0031] In the related art, existing fast classification methods often utilize preset album tags to address the need for image classification. Specifically, a limited number of predefined classification tags, such as people, flowers, and birds, are predefined. When a user wants to create a categorized album, they can search for keywords to find results for the corresponding category. However, this image classification management method requires users to consider search tags and has a relatively limited search scope. Consequently, image classification is limited, and the resulting categorized albums often fail to meet users' individual needs. Regarding image deletion, duplicate, blurry, or image-prone images, as well as images that users have lost interest in, often occupy significant storage space in long-unmanaged albums. With limited budgets, optimizing storage space requires users to spend considerable time sifting through and deleting images from their collections. This sifting process can be challenging for long-unmanaged collections. Some electronic devices offer duplicate item merging to address this issue, but intelligent filtering capabilities for other types of images remain limited. Compared to manual filtering, more convenient methods for obtaining a collection of images to be deleted include: Method 1: Narrowing the filter range through keyword searching, but the search keywords also become a limiting factor. Method 2: Filter within the categories provided by the system's intelligence, such as the screenshot categories offered by most electronic devices. However, these categories often contain a limited number of potentially deleted images. A more efficient solution is urgently needed for image deletion.

[0032] In an embodiment of the present application, for the need of image classification, the user can select at least two images in the album that need to be classified into one category. The electronic device constructs an image sequence based on the at least two images selected by the user and the images to be selected, then performs model inference on the constructed image sequence, and outputs a set of images that the user may click next. In this way, the user only needs to select a small number of images, and the electronic device can infer the user's preferences or needs based on these images and output a category of images, thereby achieving rapid classification of images. For the need of image deletion, the user can select at least two images in the album that need to be deleted. The electronic device constructs an image sequence based on the at least two images selected by the user and the images to be selected, then performs model inference on the constructed image sequence, and outputs a set of images that the user may need to delete next, thereby achieving rapid deletion of images.

[0033] The image management method provided by the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0034] Figure 1 A flowchart of an image management method provided in some embodiments of the present application is shown in FIG. Figure 1 As shown, the image management method may include the following steps 201 and 202:

[0035] Step 201: The electronic device acquires a first image sequence.

[0036] The first image sequence includes M first images selected by the user, where M is an integer greater than or equal to 2.

[0037] In some embodiments of the present application, the above-mentioned M first images may include images selected locally by the user on the electronic device, for example, images selected by the user from a photo album or a file system; or, the above-mentioned M first images may include images obtained from a server.

[0038] In some embodiments of the present application, the electronic device may determine the M first images based on the user's selection input of the M images. For example, the user may click on the M images in sequence in the album interface, and the electronic device may determine the M first images based on the user's click operation.

[0039] In some embodiments of the present application, the first image to be selected may be an image stored locally in the electronic device; or the first image to be selected may be an image obtained from a server.

[0040] In some examples, the M first images may be images selected by the user from an album, and the first to-be-selected images may be images in the album other than the images selected by the user. In some embodiments of the present application, the first to-be-selected images may include one to-be-selected image or multiple to-be-selected images.

[0041] In some embodiments of the present application, the electronic device may construct a first image sequence based on the M first images selected by the user.

[0042] In some examples, the electronic device may combine the M first images selected by the user to obtain an image click sequence, that is, a first image sequence. For example, assuming that the M first images selected by the user include n images, the image click sequence can be represented as I = {I1, I2, ..., I n}.

[0043] Step 202: The electronic device performs model inference based on the first image sequence and the N first candidate images, and outputs a target candidate image.

[0044] The target image to be selected is at least one image to be selected among the N first images to be selected, where N is a positive integer.

[0045] In some embodiments of the present application, the electronic device may perform model inference on the first image sequence and N first candidate images through a first image recommendation model, and output the above-mentioned target candidate image.

[0046] In some embodiments of the present application, the input sequence length of the first image recommendation model is a preset sequence length MAX_SEQ.

[0047] In some embodiments of the present application, the first image recommendation model includes at least an embedding layer (ie, an Embedding layer), a Transformer layer, and a multilayer perceptron (MLP).

[0048] In some embodiments of the present application, the above-mentioned Embedding layer is used to extract visual features and temporal features of images in the first image sequence to obtain an embedded representation of the features, that is, a feature set that integrates visual features and temporal features.

[0049] In some embodiments of the present application, the above-mentioned Transformer layer is used to encode the feature set output by the Embedding layer, and assign importance weights to each element in the feature set through the self-attention mechanism (self-attention) of the core module of the Transformer, model the association relationship between each image in the first image sequence, and output feature information of the fused sequence features.

[0050] In some embodiments of the present application, the MLP layer is used to perform feature extraction on the feature information output by the Transformer to further extract high-level semantics; the prediction problem is converted into a binary classification problem through the Softmax function, and the predicted probability that the n+1th picture (i.e., the first image to be selected) is the image that the user may click or select is output.

[0051] In some embodiments of the present application, the electronic device can traverse each image in the electronic device (such as a photo album), perform model inference based on the M first images selected by the user and each first candidate image traversed, and output the target candidate image.

[0052] For the convenience of description, the plurality of images to be selected may be referred to as a set of images to be selected.

[0053] In some embodiments of the present application, the electronic device can traverse the images in the album, combine at least two images selected by the user with the current image to be selected, construct an input image sequence, and then perform model inference on the input image sequence to output the target image to be selected.

[0054] In some embodiments of the present application, the electronic device may perform model reasoning on the first image sequence and the first first candidate image in the initial reasoning stage, and output the predicted probability of the first first candidate image; then, in subsequent reasoning, for each current candidate image, model reasoning is performed based on the candidate image and pre-stored sequence context information, and finally the target candidate image is output.

[0055] In some embodiments of the present application, the pre-stored sequence context information may include at least one of the following:

[0056] ① The attention parameter set generated after the first image sequence is encoded by the model, namely the key matrix (K) and the value matrix (V);

[0057] ② Embedding representation of the features of the first image sequence.

[0058] It can be understood that the embedding of the features of the first image sequence is represented as a high-dimensional feature vector.

[0059] In some examples, after the user selects at least two images that need to be managed from the album, the electronic device can, when performing inference for the first time, form an initial sequence (i.e., a first image sequence) of the M images in the order of selection, and obtain the first unselected image to be selected in the album, and then perform model inference on the initial sequence and the first image to be selected, output the predicted probability of the first image to be selected based on the inference result, and cache the K, V matrices and embedded representation of the initial sequence as sequence context information; in the subsequent processing process, when traversing and processing subsequent images to be selected in the album, the electronic device performs model inference based on the current image to be selected and the pre-stored sequence context information, and outputs the target image to be selected.

[0060] For example, when a user manages an album, he or she only needs to turn on the batch management function and click on a few photos that he or she wants to classify or delete. The electronic device can infer the user's management preferences and intentions based on the click sequence and return the Top K recommendation results for the next picture, that is, return the top K photos that the user is most likely to want to classify or delete, thereby achieving efficient management of the album.

[0061] The image management method provided by the embodiment of the present application is as follows: an electronic device obtains a first image sequence, which includes a first candidate image and M first images selected by a user; performs model inference on the above-mentioned first image sequence, and outputs a predicted probability of the first candidate image; and outputs a target candidate image based on the predicted probability of the first candidate image. In this solution, by using multiple images selected by the user as a reference, the electronic device can use model inference to quickly predict the probability of the first candidate image being selected by the user. Since the user only needs to select a small number of images, the electronic device can infer the user's preferences or needs based on these images and output the target candidate image. In this way, the user's manual image screening operation steps are simplified, and the efficiency and convenience of image management are improved.

[0062] In some embodiments of the present application, the above step 202 can be implemented by the following steps 202a and 202b.

[0063] Step 202a: The electronic device performs model inference based on the first image sequence and the N first candidate images, and calculates the prediction probabilities of the N first candidate images.

[0064] Step 202b: The electronic device outputs a target candidate image based on the predicted probabilities of the N first candidate images.

[0065] In some embodiments of the present application, the electronic device can use a trained image recommendation model to perform feature extraction and classification prediction on the first image sequence and N first candidate images, and calculate the probability value, i.e., the predicted probability, that the N first candidate images belong to a type of image clicked or selected by the user.

[0066] In some embodiments of the present application, the electronic device may traverse each image in the electronic device (such as a photo album), perform model inference based on the M first images selected by the user and each of the traversed first candidate images, calculate the predicted probability of each candidate image, and thus obtain the predicted probability of the N first candidate images. Exemplarily, the electronic device may traverse the images in the photo album, combine at least two images selected by the user with the current candidate image, construct an input image sequence, and then perform model inference on the input image sequence to calculate the predicted probability of the current candidate image, so as to obtain the predicted probability of the N first candidate images.

[0067] For the convenience of description, the plurality of images to be selected may be referred to as a set of images to be selected.

[0068] In some embodiments of the present application, for a first candidate image other than the first first candidate image among N first candidate images, the electronic device can perform feature extraction and classification prediction on the first candidate image through a trained image recommendation model, and output a probability value, i.e., a predicted probability, that the first candidate image belongs to a type of image clicked or selected by the user.

[0069] Exemplarily, the electronic device may perform model inference on the first image sequence and the first first candidate image in the initial inference stage, and calculate the predicted probability of the first candidate image; then, in subsequent inference, traverse each first candidate image except the first first candidate image, and for the first candidate image currently traversed, perform model inference based on the first candidate image and pre-stored sequence context information, and output the predicted probability of the first candidate image currently traversed.

[0070] It should be noted that, for the explanation of the pre-stored sequence context information, reference may be made to the description of the above embodiment, and to avoid repetition, it will not be described again here.

[0071] In some examples, after the user selects at least two images that need to be managed from the album, the electronic device can, when performing inference for the first time, form an initial sequence (i.e., a first image sequence) of M images in the order of selection, and obtain the first unselected image to be selected in the album, and then perform model inference on the initial sequence and the first image to be selected, output the predicted probability of the first image to be selected based on the inference result, and cache the K, V matrices and embedded representation of the initial sequence as sequence context information; in the subsequent processing process, when traversing and processing subsequent images to be selected in the album, the electronic device performs model inference based on the current image to be selected and the pre-stored sequence context information, and outputs the predicted probability of the current image to be selected.

[0072] In some examples, for the first first candidate image among N first candidate images, the electronic device can combine the first candidate image and the first image sequence to construct an input image sequence, and then input the constructed input image sequence into the first image recommendation model for processing to obtain the predicted probability of the first candidate image.

[0073] For example, assuming that the M first images selected by the user include n images, the image click sequence (ie, the first image sequence) constructed based on the n images can be expressed as I = {I1, I2, ..., I n}, the first candidate image is predicted to be the n+1th clicked image, then the input image sequence constructed based on the first candidate image and the image click sequence can be expressed as I′={I1,I2,…,I n ,I n+1}, then the input image sequence can be input into the first image recommendation model for processing to obtain the predicted probability of the first candidate image.

[0074] In some embodiments of the present application, for a first candidate image other than the first first candidate image among the N first candidate images, the electronic device may use a trained image recommendation model to perform feature extraction and classification prediction on the first candidate image, and output a probability value, i.e., a predicted probability, that the first candidate image belongs to the category of images clicked or selected by the user. For example, the input sequence length of the image recommendation model is 1.

[0075] It should be noted that in the embodiment of the present application, the entire end-side inference process includes a model for processing click sequences with an input length of MAX_SEQ and a model for processing traversal images with an input length of 1. The two models implement weighted reuse and share a set of parameters.

[0076] In some embodiments of the present application, the electronic device may select a target image that meets the requirements from a set of candidate images by probability sorting or threshold screening based on the predicted probability output by the image recommendation model.

[0077] In some embodiments of the present application, the electronic device may sort multiple images in the candidate image set in descending order of predicted probability, and select the first K images with the largest predicted probability as target images, where K is a positive integer.

[0078] In some embodiments of the present application, the above step 202a may include the following steps 202a1 to 202a3:

[0079] Step 202a1: the electronic device extracts features of each of the N first candidate images through the embedding layer of the image recommendation model to obtain feature information of the first candidate image.

[0080] Step 202a2: The electronic device generates first feature information based on the feature information of the first candidate image and the feature information of the M first images.

[0081] Step 202a3: The electronic device performs model inference on the first feature information to obtain a predicted probability of the first candidate image.

[0082] In some embodiments of the present application, the electronic device may perform feature extraction on the M first images to obtain feature information of the M first images.

[0083] In some examples, when performing inference for the first time, the electronic device may form an initial sequence (ie, a first image sequence) of the M images in a selected order, and extract feature information of the M images.

[0084] Furthermore, the electronic device may store the extracted feature information of the M images, and then in a subsequent reasoning process, generate the first feature information based on the stored feature information of the M images and the feature information of the first candidate image currently traversed.

[0085] In some embodiments of the present application, the feature information of the M first images may include visual features and time features of the M images.

[0086] It should be understood that the feature information of the M first images is the feature information of the first image sequence.

[0087] In some embodiments of the present application, the feature information of the M first images may be an embedded representation (Embedding) of the features of the M first images.

[0088] In some embodiments of the present application, the electronic device performs feature extraction on each image in the first image sequence through the Embedding layer of the first image recommendation model to obtain visual features and temporal features of the M images.

[0089] For example, for any image in the first image sequence, at the Embedding layer, visual and temporal features will be extracted as the initial representation of the image. First, the pre-trained ResNet18 network is used to extract the visual features V of the nth image in the first image sequence. n Then, use odd-even sine-cosine coding to encode the timestamp t of the picture. n Encode to obtain the time feature T n The calculation process is shown in formula (1), where T is the time period, W T represents a linear transformation, T′ n (2i) and T′ n (2i+1) represents the encoding method of even and odd dimensions respectively.

[0090]

[0091] T n =W T T′ n Formula (1)

[0092] Finally, the visual features and temporal features are added together to obtain the input E of the Transformer layer. n , this process realizes the fusion of the two encoded information and obtains the feature information of the nth image. The calculation process is shown in formula (2):

[0093] E n =T n +V n Formula (2)

[0094] In some embodiments of the present application, the electronic device performs feature extraction on the first candidate image through the Embedding layer of the first image recommendation model to obtain visual features and temporal features of the first candidate image.

[0095] It should be noted that the process of extracting the feature information of the first candidate image through the Embedding layer is the same as the process of extracting the feature information of the nth image. In order to avoid repetition, it will not be repeated here.

[0096] In some embodiments of the present application, the feature information of the first candidate image may be an embedded representation of the features of the first candidate image.

[0097] In some embodiments of the present application, the electronic device may combine the feature information of the first candidate image with the feature information of the first image sequence to obtain the first feature information.

[0098] In some embodiments of the present application, the above-mentioned first feature information can be a feature matrix composed of feature vectors of the first candidate image and M first images, that is, an embedded matrix composed of the embedded representation of the features of the first candidate image and the embedded representation of the features of the M first images.

[0099] For example, it is assumed that the feature information of the first image sequence is E={E1, E2, ..., E n}, the feature information of the first candidate image is E n+1 After the feature information of the first image sequence and the feature information of the first candidate image are combined, the first feature information obtained is E'={E1, E2, ..., E n ,E n+1}.

[0100] It should be noted that the first feature information can be used as the input of the Transformer layer, and the attention calculation is performed through the self-attention mechanism of the Transformer layer, and the new image feature representation O={O1, O2,…, O n+1}.

[0101] In some embodiments of the present application, the electronic device processes the first feature information through the self-attention mechanism of the Transformer layer and outputs the predicted probability of the first candidate image.

[0102] In an embodiment of the present application, by extracting features from the image sequence selected by the user and the image to be selected and constructing joint features, and then performing model inference on the joint features, the predicted probability of the image to be selected is output, so that a class of images that the user needs to manage in the album can be quickly determined, thereby improving processing efficiency while maintaining reasoning accuracy.

[0103] In some embodiments of the present application, the above step 202a3 may include the following steps 202b1 to 202b3:

[0104] Step 202b1: The electronic device calculates the attention parameter of the first candidate image based on the first feature information through the Transformer layer of the image recommendation model.

[0105] Step 202b2: The electronic device calculates first fusion feature information of the first candidate image and the M first images through the Transformer layer based on the attention parameter of the first candidate image and the attention parameters of the M first images.

[0106] Step 202b3: The electronic device performs information extraction processing on the first fused feature information through the MLP layer of the image recommendation model, and maps the processed first fused feature information to the probability space through the first function of the image recommendation model to obtain the predicted probability of the first candidate image.

[0107] In some embodiments of the present application, the electronic device may use the self-attention or cross-attention mechanism of the Transformer layer to perform a linear transformation on the first feature information to obtain an attention parameter set of the first image to be selected, which includes the attention parameters of the first image to be selected.

[0108] In some embodiments of the present application, the above-mentioned attention parameter set may include the query (Q) vector, key (K) vector, value (V) vector and attention weight of the first selected image, as well as the Q vector, K vector, V vector and attention weight of each first image.

[0109] Specifically, the corresponding Q, K, and V matrices are obtained by multiplying the embedded representation in the first feature information with the weight matrix of the linear mapping layer.

[0110] In some embodiments of the present application, the query (Q) vector, key (K) vector, and value (V) vector of the above-mentioned first image to be selected, and the Q vector, K vector, and V vector of each first image can constitute the Q matrix, K matrix, and V matrix corresponding to the entire input image sequence including M first images and the first image to be selected.

[0111] In some embodiments of the present application, the electronic device can use the self-attention or cross-attention mechanism of the Transformer layer to perform linear transformation on the feature information of M first images to obtain an attention parameter set of the M first images, which includes the attention parameters of the M first images.

[0112] In some embodiments of the present application, the electronic device may perform attention calculation on the above-mentioned attention parameter set through the Transformer structure of the first image recommendation model to obtain attention information of the input image sequence, thereby obtaining first fused feature information containing sequence context information.

[0113] In some examples, when performing inference for the first time, the electronic device may form an initial sequence (i.e., a first image sequence) of M images in a selected order, extract feature information of the M images, and then calculate attention parameters of the M images.

[0114] In some embodiments of the present application, the first fused feature information may be a global embedding representation of feature information of an input image sequence consisting of a click sequence and images to be selected.

[0115] Exemplarily, the electronic device applies a set of attention parameters to the first feature information, and weights feature information from different images in the first feature information according to the attention parameters to generate a discriminative feature representation.

[0116] Specifically, in the Transformer, the attention calculation process involves three matrices: query, key, and value. Attention weights are calculated by calculating the similarity between the query matrix and the key matrix. These weights are then applied to the value matrix corresponding to the key matrix. In other words, the value matrix is weighted and summed according to the attention weights, ultimately producing the weighted sum as the output. The core of this process is the scaled dot-product attention mechanism.

[0117] It should be noted that in the self-attention mechanism, each input position (or query position) generates a query vector, which can be similar to all key vectors (such as dot product) to obtain an attention weight. The attention weight reflects the correlation or similarity between the query vector and each key vector. After the attention weight is calculated, the value vector is weighted and summed by the attention weight. The result of the weighted summation is the output of the self-attention mechanism, that is, the attention information (that is, the first fused feature information), which reflects the new representation of each position in the input image sequence. This new representation has integrated the context information of the entire input image sequence.

[0118] Figure 2 A schematic diagram of the calculation process of the self-attention mechanism provided in the embodiment of this application, Figure 2 A schematic diagram of the computation process that takes the click sequence and target map as input, as shown in Figure 2 As shown in the figure, the click sequences are Item1 and Item2, and the image to be selected is Target. First, the embedded representation (Embedding1) of the feature information of the click sequence and the image to be selected is linearly transformed by the Softmax function to obtain Q, K and V. Then, the attention weight (Attention) of the internal elements of the sequence is calculated based on Q, K and V through the self-attention mechanism, and the corresponding V is weighted by the attention weight to obtain the output of the self-attention layer. Finally, the output of the self-attention layer is further processed by the feedforward neural network (FFN) to enrich the feature representation and obtain the fused feature information (Embedding2).

[0119] In some embodiments of the present application, the electronic device may store the attention parameters of the M first images and the feature information of the M first images.

[0120] In some embodiments of the present application, the electronic device may store the attention parameters of the M first images and the feature information of the M first images in a cache area of the electronic device.

[0121] In some examples, the electronic device can store the calculated attention parameters of M images, and then in the subsequent reasoning process, through the attention mechanism of the Transformer layer, operate on the attention parameter set of the stored M images and the attention parameter set of the first candidate image currently traversed to obtain the fusion feature information of the first candidate image and the M first images.

[0122] Combined with the above Figure 2 , Figure 3 This is a schematic diagram of the calculation process after deleting the redundant calculation of the click sequence, as shown in Figure 3 As shown in the figure, during the first inference, the K, V, and Embedding results of the click sequence are cached. When processing subsequent candidate images, only the current candidate image is needed as input to calculate its Q, K, and V. Then, the cached click sequence K, V, and Embedding are used in combination with the Q of the current image for calculation.

[0123] It should be noted that due to the existence of the sequence mask, the elements after the current position will not affect the calculation results of Embedding2. In addition, the cache mechanism avoids the repeated calculation of Q, K, and V of the click sequence each time, thereby improving the calculation efficiency.

[0124] In some embodiments of the present application, the electronic device may further perform feature extraction on the first fused feature information through the MLP layer of the first image recommendation model, such as dimensionality reduction, feature selection or feature fusion, to extract key feature information useful for prediction, thereby improving the accuracy of subsequent predictions.

[0125] In some embodiments of the present application, the first function may be a Softmax function.

[0126] In some embodiments of the present application, the electronic device maps the processed first fused feature information to a probability space through the Softmax function of the first image recommendation model, and calculates the probability that the first candidate image belongs to a category of images selected by the user.

[0127] The following schematically illustrates the process of the above-mentioned image management method in combination with the model structure of the image recommendation model provided in the embodiment of the present application.

[0128] like Figure 4As shown, the user click sequence data is used as input, for example: Item1, Item2, Item3, and the target image (i.e., the first image to be selected) is Item4, which is the image that the user may click next. First, in the Embedding layer, the pre-trained ResNet18 network is used to extract the visual features V_1, V_2, V_3 of Item1, Item2, and Item3. Then, the timestamps t_1, t_2, t_3 of Item1, Item2, and Item3 are encoded using odd-even sine-cosine coding to obtain the temporal features T_1, T_2, and T_3. Then, according to the above formulas (1) and 2, the visual features and temporal features are added to obtain the input E_1, E_2, and E_3 of the Transformer layer. In the Transformer layer, the obtained click sequence feature set {E_1, E_2, E_3} is masked with a sequence mask to cover the features after the kth image to ensure that the prediction depends on the previous image. The self-attention mechanism assigns importance weights to elements in the sequence, modeling the relationships between the images in the sequence. After encoding through the Transformer, a new image representation {O_1, O_2, O_3, O_4} is output that incorporates the sequence features. The MLP layer further extracts features from the Transformer output {O_1, O_2, O_3, O_4}. Finally, the Softmax function is used to transform the prediction problem into a binary classification problem, outputting the probability that the n+1th image (i.e., Item 4) is the target click image.

[0129] In this way, by combining visual and temporal features, the Transformer model is used to model the sequence, and feature extraction and probability output are performed through the MLP and Softmax layers, finally predicting the picture that the user may click next.

[0130] In an embodiment of the present application, feature information of multiple images selected by the user and the candidate images to be predicted is processed through an attention mechanism, the correlation between the images is modeled, a new image feature representation that incorporates sequence features is output, and the probability of the candidate image being selected is predicted based on the new image feature representation, thereby being able to quickly and accurately determine the image the user wants to select from a large number of images.

[0131] In combination with the above embodiments, the following schematically illustrates the process of the image management method provided in the embodiments of the present application. For example, the image management method may include the following steps:

[0132] Step 11: The M images clicked by the user are formed into an initial sequence in the order of selection, and the first unselected image to be selected in the album (such as the first image to be selected) is obtained.

[0133] Step 12: Perform joint feature encoding on the initial sequence and the first image to be selected, and calculate Q, K, and V of the initial sequence and the first image to be selected through the self-attention mechanism.

[0134] Step 13: Generate sequence global embedding representation (Embedding2).

[0135] Step 14: Output the predicted probability of the first candidate image based on the sequence global embedding representation.

[0136] Step 15: Cache K, V matrix and Embedding2 as sequence context information.

[0137] When traversing and processing subsequent images to be selected in the album, the electronic device performs the following operations:

[0138] Step 16: Input the current image to be selected (such as the second image to be selected) into the feature extraction module to generate an image feature vector.

[0139] Step 17: Based on the pre-stored sequence context information (K, V, Embedding2) and the current image features, the model is inferred through the attention mechanism.

[0140] Step 18: Output the predicted probability of the current candidate image.

[0141] It should be noted that when using the Transformer's self-attention mechanism, if the complete click sequence is used as the model input during each inference process, the Q, K, and V of the click sequence will be repeatedly calculated each time, resulting in unnecessary resource waste. Figure 2 As shown in the first row, due to the existence of the sequence mask, elements after the current position will not affect the calculation result of Embedding2. Therefore, the K, V and Embedding2 of the click sequence can be directly cached, thereby improving calculation efficiency.

[0142] In an embodiment of the present application, when the model performs inference for the first time, it takes the complete click sequence and the first image to be selected as input and outputs the predicted probability of the first image. At this time, the calculated results of K, V and Embedding2 of the click sequence can be cached for subsequent inference. When processing subsequent images to be selected, it is only necessary to use the current image to be selected as input, calculate its Q, K, V, and use the cached click sequence K, V and Embedding2, combined with the image Q for calculation. In this way, the inference results of subsequent images to be selected can be obtained directly, thereby improving the overall computing efficiency.

[0143] In some embodiments of the present application, before step 202, the image management method may further include the following steps 209 and 210:

[0144] Step 209: The electronic device obtains the sequence length of the first image sequence.

[0145] Step 210: When the sequence length is less than the preset sequence length, the electronic device performs a padding operation on the first image sequence and performs model inference on the padded first image sequence; or, when the sequence length is greater than the preset sequence length, the electronic device performs a truncation operation on the first image sequence and performs model inference on at least two obtained subsequences.

[0146] In some embodiments of the present application, the preset sequence length may be a fixed maximum click sequence length MAX_SEQ.

[0147] In some embodiments of the present application, when the sequence length is less than a preset sequence length, the electronic device performs a padding operation on the end of the first image sequence until the sequence length meets the preset sequence length.

[0148] Exemplarily, when the sequence length of the first image sequence is less than MAX_SEQ, 0 is padded at the end of the sequence of the first image sequence, so that the sequence length of the padded first image sequence is equal to MAX_SEQ.

[0149] In some embodiments of the present application, when the sequence length is greater than a preset sequence length, the electronic device truncates the first image sequence into N subsequences according to the preset sequence length. Furthermore, a padding operation is performed on subsequences whose sequence length is less than the preset sequence length so that the sequence length of each subsequence meets the preset sequence length.

[0150] Exemplarily, when the sequence length of the first image sequence exceeds MAX_SEQ, the first image sequence is truncated into N subsequences in units of MAX_SEQ, and a padding operation is performed on the last subsequence that is less than MAX_SEQ.

[0151] Furthermore, the electronic device may store the K, V, and Embedding values of each subsequence for use in reasoning about the next subsequence.

[0152] In some embodiments of the present application, the electronic device may process the at least two obtained subsequences separately to obtain attention parameter sets for the at least two subsequences. Subsequently, the attention parameter sets for the at least two subsequences are merged to obtain an attention parameter set for the first image sequence. The electronic device may then perform subsequent model inference on the feature information of the first image sequence and output a predicted probability for the first candidate image.

[0153] It's important to note that click sequences can be fed into the model in batches during inference, accelerating inference. However, due to limitations in client-side hardware resources and inference efficiency, inference models often require fixed-length input, while the number of user clicks is often variable.

[0154] In an embodiment of the present application, the maximum click sequence length MAX_SEQ is fixed. When the click sequence length is less than MAX_SEQ, a padding operation is performed at the end of the sequence until the sequence length meets MAX_SEQ. Due to the existence of the sequence mask, the sequence of the padding part will not affect the encoding result of the real click sequence. When the sequence length exceeds MAX_SEQ, the sequence is truncated into N subsequences in units of MAX_SEQ, and the inference is performed in sequence, and the K, V, and Embedding values of each subsequence are stored for the inference of the next subsequence. Similarly, a padding operation is performed on the last subsequence that is less than MAX_SEQ. In this way, by padding or truncating the image sequence, image sequences of different lengths can meet the input requirements of the model, thereby improving the compatibility of the model with different data sets.

[0155] In some embodiments of the present application, starting from traversing the second image to be selected (i.e., the second image to be selected), the model input is fixed to 1, and the pre-order cache value is used for inference calculation. Therefore, the entire end-side inference process has a model for processing click sequences with an input length of MAX_SEQ and a model for processing traversed images with an input length of 1. The two models implement weight reuse and share a set of parameters. Combined with the click sequence batch processing requirements, the K, V, Embedding cache design, such as Figure 5 As shown in the figure, define a cache of sufficient length and use pointers to mark the start and end positions of the current cache. To ensure a fixed cache length for the input model, keep the interval between start and end at max_seq_len, with end pointing to the end of the sequence. When the click sequence exceeds MAX_SEQ and needs to be divided into multiple subsequences for inference, the K, V, and Embedding values generated by the most recent inference are added to the cache, and the pointer position is adjusted.

[0156] In some embodiments of the present application, when using cached data for inference, the sequence mask needs to be modified accordingly, such as Figure 6 As shown, the empty cache (cache_pad) is treated as -inf, thereby masking the impact of the empty cache.

[0157] In some embodiments of the present application, the image management method may further include the following steps 211 and 212:

[0158] Step 211: The electronic device obtains model data of the image recommendation model.

[0159] The above-mentioned model data includes model weight parameters and activation parameters, and the image recommendation model is used to perform model inference based on the first image sequence and N first candidate images.

[0160] Step 212: The electronic device quantizes the model data using the target quantization accuracy to obtain quantized model data.

[0161] The target quantitative accuracy is determined based on the importance of the model data.

[0162] In some embodiments of the present application, the electronic device may evaluate the importance of the model weight parameter; use a higher quantization precision for a model weight parameter with high importance; or use a lower quantization precision for a model weight parameter with low importance.

[0163] It's important to note that in a neural network model, weight parameters are the coefficients connecting different neurons and determine the strength of signal transmission within the network. By learning from a large amount of training data, the model continuously adjusts the weight parameters, enabling it to better fit the training data and improve prediction accuracy. The activation parameter generally refers to the output value of a neuron after weighted summation and processing through the activation function.

[0164] In some embodiments of the present application, the target quantization accuracy may be 8-bit quantization, 16-bit quantization, or the like.

[0165] In some embodiments of the present application, the electronic device can determine the importance by analyzing factors such as the size, distribution, and role of the weight parameters in the model. For example, the weight parameters that play a key role in the model (such as the weight parameters connecting important feature layers) are considered to be of high importance; while some smaller weight parameters that have less impact on model performance are considered to be of low importance. Similarly, some activation parameters that play a key role in the model are considered to be of high importance; while some smaller activation parameters that have less impact on model performance are considered to be of low importance.

[0166] In some embodiments of the present application, the electronic device may select different quantization precisions for weight parameters of different importance based on the evaluation results. For weight parameters of high importance, a higher quantization precision (such as 16 bits) is used; for weight parameters of low importance, a lower quantization precision (such as 8 bits) is used.

[0167] It should be noted that quantization refers to converting floating-point parameters in a model into low-precision integer representations to reduce the model's storage space and computational complexity. Higher quantization precision preserves more information but increases storage and computational costs; lower quantization precision reduces storage and computational costs but may result in reduced model performance. However, the deployment of on-device models is constrained by electronic device resource limitations and real-time requirements, necessitating model compression to streamline the model and reduce storage and computational requirements.

[0168] In some embodiments of the present application, electronic devices use quantization technology to reduce the storage space and computing resource requirements of the feature maps of the weight and activation models. Hybrid precision quantization is introduced. For example, for more important attention mechanism modules, higher precision (such as FP16) is retained, while non-critical fully connected modules use low-precision (INT8 / INT4) compression. This reduces memory usage and computational overhead while ensuring model accuracy as much as possible, achieving a synergistic improvement in computational efficiency and model accuracy.

[0169] In an embodiment of the present application, by selecting different quantization precisions according to the importance of the model parameters, the performance of the model can be maintained while reducing storage and computing costs. A higher quantization precision is used for model parameters with high importance to ensure that key information of the model is not lost, thereby ensuring the accuracy of the recommendation results. In this way, by quantizing the model data of the image recommendation model and selecting appropriate quantization precision according to the importance, the performance of the model can be maintained while reducing storage costs and improving computing efficiency, facilitating the deployment and application of the model on electronic devices.

[0170] In the embodiment of the present application, deploying the image recommendation model on the terminal device can not only reduce the dependence on cloud services and reduce system latency, thereby improving user experience, but also effectively protect user privacy data.

[0171] Figure 7 A flow chart of a model training method provided in some embodiments of the present application, which can be applied to a server, such as Figure 7 As shown, the model training method may include the following steps 301 and 302:

[0172] Step 301: The server obtains a first image sample set and a second image sample set.

[0173] The first image sample set includes images to be selected and M images selected by the user, and the second image sample set includes random images and M images selected by the user. The random images are generated or obtained based on predetermined rules.

[0174] In some embodiments of the present application, the first image sample set may include at least one image sequence, and each image sequence includes at least two images clicked by the user.

[0175] In some embodiments of the present application, the second image sample set may include at least one image sequence, and each image sequence includes at least two images clicked by the user and a random image.

[0176] Step 302: The server trains the image recommendation model based on the first image sample set and the second image sample set to obtain model parameters of the image recommendation model.

[0177] In some embodiments of the present application, the server takes the first image sample set and the second image sample set as input, and trains the image recommendation model through a specific algorithm so that the image recommendation model learns the user's image selection patterns and preferences, so as to be able to more accurately recommend images that the user may be interested in.

[0178] In some embodiments of the present application, the image recommendation model is trained to adjust the model parameters to minimize the loss function, and an optimized image recommendation model is finally obtained.

[0179] In some embodiments of the present application, the optimized image recommendation model may include a model parameter file and a model structure. Furthermore, the model parameter file may include all optimized weights and biases, and the model structure may be defined by code.

[0180] In some embodiments of the present application, the above-mentioned model parameters may include model weights, biases, etc.

[0181] In some embodiments of the present application, after obtaining a trained image recommendation model, the server sends the trained model parameters to the electronic device, and the electronic device can directly load the model parameters into the model structure and perform inference.

[0182] In some embodiments of the present application, the electronic device may load the model parameters into the model structure code of the first image recommendation model and the model structure code of the second image recommendation model respectively.

[0183] In an embodiment of the present application, an image recommendation model is trained based on a dataset containing images actively selected by the user and a dataset containing images selected by the user and random images, thereby combining positive and negative samples for user preference understanding training, so that the model can learn the temporal characteristics of user click behavior (such as the tendency to continuously click on similar images), thereby enabling the model to more comprehensively understand user interests and improve the accuracy of recommended images.

[0184] In some embodiments of the present application, the above step 302 may include the following steps 302a to 302d:

[0185] Step 302a: The server performs feature mapping on the first image sample set and the second image sample set respectively through the embedding layer of the image recommendation model to obtain corresponding image feature information.

[0186] Step 302b: The server uses the Transformer layer of the image recommendation model to respectively perform weight distribution and fusion on the feature information of the first image sample set and the feature information of the second image sample set to obtain corresponding fused feature information.

[0187] Step 302c: The server calculates the prediction probabilities of the candidate image and the random image respectively based on the corresponding fusion feature information.

[0188] Step 302d: The server adjusts the weight parameters of the image recommendation model based on the predicted probability using a cross-entropy loss function to obtain model parameters of the image recommendation model.

[0189] In some embodiments of the present application, the original image data in the first and second image sample sets is converted into embedded representations, i.e., high-dimensional feature vectors, by the embedding layer of the image recommendation model. For example, by processing the input first and second image sample sets, two sets of image feature information are output, each set of feature information being a set of feature vectors for all images in the sample set.

[0190] In some embodiments of the present application, the server uses the Transformer module of the image recommendation model to weight and fuse the two sets of feature information, and outputs two sets of fused feature information, where each set of fused feature information is a weighted combination of image features in the sample set.

[0191] It should be noted that the explanation of the image recommendation model can be found in the description of the above-mentioned electronic device side method embodiment, and will not be repeated here.

[0192] In some embodiments of the present application, the server calculates the predicted probabilities of the candidate image and the random image based on the two sets of fused feature information. Specifically, the fused features are mapped to the category space of the candidate image (such as image ID or label), and then the output is normalized into a probability distribution using a Softmax function, representing the likelihood of the user selecting each image.

[0193] In some embodiments of the present application, the server adjusts the weight parameters of the image recommendation model based on the difference between the predicted probability and the true label (i.e., the image actually selected by the user) through the cross-entropy loss function to minimize the loss function and obtain optimized model parameters (such as the weight matrix and bias vector).

[0194] In this way, by iteratively optimizing the model parameters, the model can more accurately predict the probability of users selecting images, thereby improving the recommendation performance.

[0195] The following is an illustrative example of the image recommendation model training process.

[0196] For example, the model takes the sequence of pictures clicked by the user as input, and outputs the set of pictures that the user is likely to click next. The specific steps of model training include:

[0197] Step 1: Training dataset generation.

[0198] It should be noted that there is often a correlation between images that users click on consecutively within a short period of time. This data can reflect the user's focus on the images during this period. Therefore, this paper collects the operation sequence data when users click on images in batches to generate a positive sample dataset of user click sequences. At the same time, a negative sample dataset is generated by randomly replacing the last image in the sequence.

[0199] Step 2: Build the model structure.

[0200] It should be noted that this application proposes an image recommendation model, which can be used to select the image according to the image click sequence I = {I1, I2, ..., I n}, predict the probability S of a certain picture becoming the n+1th clicked picture n+1 . That is S n+1 =Model(I1,I2,…,I n ,I n+1 ). The model mainly consists of three layers: Embedding, Transformer and MLP.

[0201] In the album pictures, the picture content and time are two key information. In the Embedding layer, visual and temporal features will be extracted as the initial representation of the picture. First, the pre-trained ResNet18 network is used to extract the visual features V of the nth picture in the sequence. n Then, use odd-even sine-cosine coding to encode the timestamp t of the picture. n Encode to obtain the time feature T n , as shown in the above formula (1). Where T is the time period, W T represents a linear transformation, T′ n (2i) and T′ n (2i+1) represents the encoding method of even and odd dimensions respectively.

[0202] Finally, the visual features and temporal features are added together to obtain the input E of the Transformer modulen , this process realizes the information fusion of the two codes, as shown in the above formula (2).

[0203] The Transformer model has become an ideal choice for handling complex recommendation tasks due to its advantages in sequence modeling, especially its excellent ability to capture long-distance dependencies and flexible context attention mechanism. Therefore, the present invention adopts the Transformer structure to encode sequence features. During model training, the feature set E of the click and target images obtained by the Embedding layer is E = {E1, E2, ..., E n+1} as the module input. A sequence mask is used to mask the features after the kth image, ensuring that the prediction depends on the previous image. Next, the self-attention mechanism, a core module of the Transformer, is used to assign importance weights to the elements in the sequence and model the association between the sequence images, as shown in formula (3):

[0204]

[0205] Where Q = XW Q , K=XW K , V=XW V , represent query, key, and value respectively, and X is the input of self-attention. The output of the self-attention layer is passed through a feed-forward neural network (FFN), which introduces nonlinear transformation and enriches the feature representation. After the Transformer encoding process, the mutual influence between sequence elements will be modeled, and the final output is a new image feature representation O = {O1, O2, ..., O n+1}.

[0206] The output of the Transformer is further extracted through the MLP layer. Finally, the prediction problem is converted into a binary classification problem using the Softmax function, outputting the probability that the n+1th image is the target clicked image. The training process uses the cross-entropy loss function, as shown in Formula (4), where p(x) is the probability that image x is predicted to be the next clicked image, and y∈{0,1} is the positive and negative sample labels.

[0207]

[0208] In combination with the above embodiments, the main processes of model training, terminal-side deployment, and model reasoning provided in the embodiments of this application are exemplified below.

[0209] Step 21: The server builds a picture click sequence dataset.

[0210] Exemplarily, a sequence of pictures clicked by users is collected to form a data set.

[0211] Step 22: The server trains the image recommendation model to model the user's screening intention.

[0212] Exemplarily, the server uses the collected image click sequence dataset to train an image recommendation model to identify the user's image screening intention.

[0213] Step 23: Deploy the image recommendation model on the electronic device.

[0214] Exemplarily, the server sends the model parameters of the trained image recommendation model and the corresponding model structure file to the electronic device, and the electronic device loads the model parameters and the corresponding model structure file.

[0215] Step 24: After the user clicks to generate the image sequence, the electronic device activates the intelligent recommendation function.

[0216] For example, in actual use, the user generates a new picture sequence by clicking on a picture and starts the smart recommendation function.

[0217] Step 25: The electronic device traverses the album image collection and uses the image recommendation model to filter out images related to the sequence TopK.

[0218] For example, after the intelligent recommendation function is started, the user's album picture collection is traversed to filter out the top K pictures that are most relevant to the current picture sequence.

[0219] Step 26: The electronic device quickly manages the photo album based on the screening results.

[0220] Illustratively, the electronic device quickly performs image classification or deletion operations based on the screened related pictures.

[0221] The above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of this application do not limit this.

[0222] The image management method provided in the embodiment of the present application can be executed by an image management device. In the embodiment of the present application, the image management device provided in the embodiment of the present application is described by taking the image management device executing the image management method as an example.

[0223] Figure 8 A schematic diagram of the structure of the image management device provided in the embodiment of the present application is shown as follows: Figure 8As shown, the image management device 600 may include a processing module 601, which is used to: obtain a first image sequence, the first image sequence includes M first images selected by the user, M is an integer greater than or equal to 2; perform model inference based on the first image sequence and N first candidate images, and output a target candidate image, which is at least one candidate image among the above-mentioned N first candidate images, and N is a positive integer.

[0224] In some embodiments of the present application, the above-mentioned processing module is specifically used to: perform model inference based on the first image sequence and N first candidate images, and calculate the prediction probability of the N first candidate images; and output the target candidate image based on the prediction probability of the N first candidate images.

[0225] In some embodiments of the present application, the above-mentioned processing module is specifically used to: for each first candidate image among N first candidate images, extract features of the first candidate image through the embedding layer of the image recommendation model to obtain feature information of the first candidate image; generate first feature information based on the feature information of the first candidate image and the feature information of the M first images; perform model inference on the first feature information to obtain the predicted probability of the first candidate image.

[0226] In some embodiments of the present application, the above-mentioned processing module is specifically used to: calculate the attention parameter of the first candidate image based on the first feature information through the Transformer layer of the image recommendation model; calculate the first fused feature information of the first candidate image and the M first images based on the attention parameter of the first candidate image and the attention parameters of the M first images through the Transformer layer; perform information extraction processing on the first fused feature information through the MLP layer of the image recommendation model, and map the processed first fused feature information to the probability space through the first function of the image recommendation model to obtain the predicted probability of the first candidate image.

[0227] In some embodiments of the present application, the above-mentioned processing module is also used to: obtain the sequence length of the first image sequence before performing model inference based on the first image sequence and N first candidate images and outputting the target candidate image; when the sequence length is less than the preset sequence length, perform a padding operation on the first image sequence, and perform model inference on the padded first image sequence; or, when the sequence length is greater than the preset sequence length, perform a truncation operation on the first image sequence, and perform model inference on at least two obtained subsequences.

[0228] In some embodiments of the present application, the above-mentioned processing module is also used to: obtain model data of the image recommendation model, the model data including model weight parameters and activation parameters, and the above-mentioned image recommendation model is used to perform model inference based on the first image sequence and N first selected images; use the target quantization accuracy to quantize the model data to obtain quantized model data, and the target quantization accuracy is determined based on the importance of the model data.

[0229] An image management device provided in an embodiment of the present application obtains a first image sequence comprising a first candidate image and M first images selected by a user; performs model inference on the first image sequence and outputs a target candidate image. In this solution, using multiple images selected by the user as references, the image management device can use model inference to quickly predict the target image the user intends to select. Since the user only needs to select a small number of images, the device can infer the user's preferences or needs based on these images and output the target candidate image. This simplifies the manual image screening process for users and improves the efficiency and convenience of image management.

[0230] The image management device in the embodiments of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, an in-vehicle electronic device, a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiments of the present application do not specifically limit it.

[0231] The image management device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0232] The image management device provided in the embodiment of the present application can implement each process implemented in the embodiment of the image management method. To avoid repetition, they are not described here.

[0233] The model training method provided in the embodiment of the present application can be executed by a model training device. In the embodiment of the present application, the model training device is used as an example to illustrate the model training method.

[0234] Figure 9 A schematic diagram of the structure of the model training device provided in the embodiment of the present application is shown as follows: Figure 9 As shown, the model training device 700 may include a processing module 701, which is used to: obtain a first image sample set and a second image sample set, the first image sample set includes images to be selected and M images selected by the user, and the second image sample set includes random images and M images selected by the user, and the random images are generated or obtained based on predetermined rules; train the image recommendation model on the first image sample set and the second image sample set to obtain model parameters of the image recommendation model.

[0235] In some embodiments of the present application, the above-mentioned processing module is specifically used to: perform feature mapping on the first image sample set and the second image sample set respectively through the embedding layer of the image recommendation model to obtain corresponding image feature information; perform weight assignment and fusion on the feature information of the first image sample set and the feature information of the second image sample set respectively through the Transformer layer of the image recommendation model to obtain corresponding fused feature information; calculate the prediction probabilities of the selected images and the random images respectively based on the corresponding fused feature information; adjust the weight parameters of the image recommendation model based on the predicted probability through the cross-entropy loss function to obtain the model parameters of the image recommendation model.

[0236] The model training device provided in the embodiment of the present application trains an image recommendation model based on a dataset containing images actively selected by the user and a dataset containing images selected by the user and random images, thereby combining positive samples and negative samples to perform user preference understanding training, so that the model can learn the temporal characteristics of user click behavior (such as the tendency to continuously click on similar images), thereby enabling the model to more comprehensively understand user interests and improve the accuracy of recommended images.

[0237] The model training device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0238] The model training device in the embodiment of the present application can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0239] The model training device provided in the embodiment of the present application can implement each process implemented in the embodiment of the model training method. To avoid repetition, it will not be described here.

[0240] Alternatively, as Figure 10 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores a program or instruction that can be run on the processor 801, and when the program or instruction is executed by the processor 801, the various steps of the above-mentioned image management method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0241] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0242] Figure 11 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0243] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101 , a network module 102 , an audio output unit 103 , an input unit 104 , a sensor 105 , a display unit 106 , a user input unit 107 , an interface unit 108 , a memory 109 , and a processor 110 .

[0244] Those skilled in the art will understand that the electronic device 100 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0245] In which, the processor 110 is used to: obtain a first image sequence, the first image sequence includes M first images selected by the user, M is an integer greater than or equal to 2; perform model inference based on the first image sequence and the first candidate image, and output a target candidate image, which is at least one candidate image among the above-mentioned N first candidate images, and N is a positive integer.

[0246] In some embodiments of the present application, the above-mentioned processor 110 is specifically used to: perform model inference based on the first image sequence and N first candidate images, and calculate the prediction probability of the N first candidate images; and output the target candidate image based on the prediction probability of the N first candidate images.

[0247] In some embodiments of the present application, the above-mentioned processing module is specifically used to: for each first candidate image among N first candidate images, extract features of the first candidate image through the embedding layer of the image recommendation model to obtain feature information of the first candidate image; generate first feature information based on the feature information of the first candidate image and the feature information of the M first images; perform model inference on the first feature information to obtain the predicted probability of the first candidate image.

[0248] In some embodiments of the present application, the above-mentioned processor 110 is specifically used to: calculate the attention parameter of the first candidate image based on the first feature information through the Transformer layer of the image recommendation model; calculate the first fused feature information of the first candidate image and the M first images based on the attention parameter of the first candidate image and the attention parameters of the M first images through the Transformer layer; perform information extraction processing on the first fused feature information through the MLP layer of the image recommendation model, and map the processed first fused feature information to the probability space through the first function of the image recommendation model to obtain the predicted probability of the first candidate image.

[0249] In some embodiments of the present application, the above-mentioned processor 110 is further used to: obtain the sequence length of the first image sequence before performing model inference based on the first image sequence and N first candidate images and outputting the target candidate image; when the sequence length is less than the preset sequence length, perform a padding operation on the first image sequence and perform model inference on the padded first image sequence; or, when the sequence length is greater than the preset sequence length, perform a truncation operation on the first image sequence and perform model inference on at least two obtained subsequences.

[0250] In some embodiments of the present application, the processor 110 is further used to: obtain model data of an image recommendation model, the model data including model weight parameters and activation parameters, the image recommendation model being used to perform model inference based on a first image sequence and N first selected images; quantize the model data using a target quantization accuracy to obtain quantized model data, and the target quantization accuracy is determined based on the importance of the model data.

[0251] The image management device provided in an embodiment of the present application comprises an electronic device acquiring a first image sequence, the first image sequence comprising a first candidate image and M first images selected by a user; performing model inference on the first image sequence to output a predicted probability of the first candidate image; and outputting a target candidate image based on the predicted probability of the first candidate image. In this solution, by using multiple images selected by the user as a reference, the electronic device can use model inference to quickly predict the target image the user wants to select. Since the user only needs to select a small number of images, the user's preferences or needs can be inferred based on these images and the target candidate image can be output. This simplifies the steps for the user to manually screen images, improving the efficiency and convenience of image management.

[0252] It should be understood that in an embodiment of the present application, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0253] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0254] Processor 110 may include one or more processing units. Optionally, processor 110 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 110.

[0255] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment or the various processes of the above-mentioned model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0256] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk. Alternatively, the processor is the processor in the server described in the above embodiment.

[0257] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image management method embodiment, or to implement the various processes of the above-mentioned model training method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0258] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0259] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image management method embodiment or the various processes of the above-mentioned model processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0260] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0261] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0262] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image management method, executed by an electronic device, characterized in that: The method comprises: Acquire a first image sequence, where the first image sequence includes M first images selected by a user, where M is an integer greater than or equal to 2; Model inference is performed based on the first image sequence and N first candidate images, and a target candidate image is output. The target candidate image is at least one candidate image among the N first candidate images, where N is a positive integer.

2. The method according to claim 1, characterized in that The performing model inference based on the first image sequence and the N first candidate images to output a target candidate image includes: Performing model inference based on the first image sequence and the N first candidate images to calculate prediction probabilities of the N first candidate images; Based on the predicted probabilities of the N first candidate images, the target candidate image is output.

3. The method according to claim 2, characterized in that The performing model inference based on the first image sequence and the N first candidate images to calculate the prediction probabilities of the N first candidate images includes: For each of the N first candidate images, extract features of the first candidate image using an embedding layer of an image recommendation model to obtain feature information of the first candidate image; generating first feature information based on the feature information of the first candidate image and the feature information of the M first images; Model inference is performed on the first feature information to obtain a predicted probability of the first candidate image.

4. The method according to claim 3, characterized in that The performing model inference on the first feature information to obtain a predicted probability of the first candidate image includes: calculating, by a Transformer layer of the image recommendation model, an attention parameter of the first candidate image based on the first feature information; Calculating, through the Transformer layer, first fused feature information of the first candidate image and the M first images based on the attention parameter of the first candidate image and the attention parameters of the M first images; The first fused feature information is subjected to information extraction processing through the MLP layer of the image recommendation model, and the processed first fused feature information is mapped to a probability space through the first function of the image recommendation model to obtain a predicted probability of the first candidate image.

5. The method according to claim 1, wherein Before performing model inference based on the first image sequence and the N first candidate images and outputting the target candidate image, the method further includes: Obtaining a sequence length of the first image sequence; When the sequence length is less than a preset sequence length, a padding operation is performed on the first image sequence, and model inference is performed on the padded first image sequence; or, when the sequence length is greater than the preset sequence length, a truncation operation is performed on the first image sequence, and model inference is performed on at least two obtained subsequences.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining model data of an image recommendation model, the model data including model weight parameters and activation parameters, the image recommendation model being used to perform model inference based on the first image sequence and the N first candidate images; The model data is quantized using a target quantization accuracy to obtain quantized model data, wherein the target quantization accuracy is determined based on the importance of the model data.

7. A model training method, executed by a server, characterized in that: The method comprises: Acquire a first image sample set and a second image sample set, wherein the first image sample set includes images to be selected and M images selected by a user, and the second image sample set includes random images and M images selected by the user, wherein the random images are generated or acquired based on a predetermined rule; An image recommendation model is trained based on the first image sample set and the second image sample set to obtain model parameters of the image recommendation model.

8. The method according to claim 7, characterized in that The training of the image recommendation model based on the first image sample set and the second image sample set to obtain model parameters of the image recommendation model includes: Performing feature mapping on the first image sample set and the second image sample set respectively through the embedding layer of the image recommendation model to obtain corresponding image feature information; performing weight distribution and fusion on the feature information of the first image sample set and the feature information of the second image sample set through the Transformer layer of the image recommendation model to obtain corresponding fused feature information; Based on the corresponding fusion feature information, respectively calculating the prediction probabilities of the candidate image and the random image; The weight parameters of the image recommendation model are adjusted based on the predicted probability through a cross entropy loss function to obtain model parameters of the image recommendation model.

9. An image management device, characterized in that: The device includes a processing module, wherein the processing module is configured to: Acquire a first image sequence, where the first image sequence includes M first images selected by a user, where M is an integer greater than or equal to 2; Model inference is performed based on the first image sequence and the first candidate image, and a target candidate image is output. The target candidate image is at least one candidate image among the N first candidate images, where N is a positive integer.

10. An electronic device, characterized in that: The image management method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image management method according to any one of claims 1 to 6 are implemented.

11. A server, characterized in that: It includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the model training method as described in claim 7 or 8 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, it implements the steps of the file management method as described in any one of claims 1 to 6, or implements the steps of the model training method as described in claim 7 or 8.

Images