Image evaluation and image generation method and device based on image evaluation, medium and computer program product

Abstract

The invention relates to an image evaluation method, which is used for electronic equipment, and is characterized in that the method comprises the following steps: receiving an image; by using an image evaluation model, learning features of multiple scales of the image, determining a fusion weight of the features of the multiple scales through an attention mechanism, fusing the features of the multiple scales based on the fusion weight to obtain a fused image feature, determining and outputting a score of the image based on the fused image feature, and receiving interaction of a user to the score; and adjusting parameters of the image evaluation model based on interaction and historical interaction data of the user. The invention also relates to an image generation method based on image evaluation, as well as to image evaluation and an image generation device based on image evaluation, a medium and a computer program product.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image evaluation and an image generation method, device, medium and computer program product based on the image evaluation. Background Art

[0002] With the rapid development of the information age, generative artificial intelligence has been widely used in the visual and media industries to improve image productivity and quality. At the same time, the new retail industry also uses generative images to meet personalized needs by analyzing factors such as user demographics, purchase history, and geographic location. However, existing technologies have obvious shortcomings in generating image content that meets corporate brand characteristics.

[0003] In the existing technology, many methods fail to fully consider the brand image of a specific enterprise, resulting in the generated images not accurately reflecting the values ​​of the enterprise. In addition, existing aesthetic evaluation methods often lose important aesthetic information due to the limitation of input image size, which further affects the quality of image generation. Summary of the invention

[0004] The purpose of the present invention is to provide an image evaluation and an image generation method, device, medium and computer program product based on image evaluation, so as to solve the technical problem that the aesthetic evaluation model in the prior art cannot generate images that meet the characteristics of the corporate brand.

[0005] A first embodiment of the present invention discloses an image evaluation method for an electronic device, the method comprising:

[0006] receiving an image;

[0007] Evaluate the model using images:

[0008] Learning features of the image at multiple scales,

[0009] The fusion weights of the features of the multiple scales are determined by the attention mechanism.

[0010] The features of the multiple scales are fused based on the fusion weight to obtain a fused image feature,

[0011] Based on the fused image features, determining and outputting a score of the image,

[0012] receiving user interactions with the ratings;

[0013] Based on the interaction and the user's historical interaction data, parameters of the image evaluation model are adjusted.

[0014] According to the method of the first embodiment of the present invention,

[0015] The method further includes: receiving interaction data of the user, the interaction data including the number of likes and the content of comments; and learning the features of the interaction data using the image evaluation model;

[0016] The determining and outputting the score of the image based on the fused image features further includes: determining and outputting the score of the image based on the fused image features and features of the interaction data.

[0017] According to the method of the first embodiment of the present invention, the image evaluation model includes an image feature module, and the image feature module includes:

[0018] A fine-grained branch that learns fine-scale features of the image, including texture and color distribution in the image;

[0019] A meso-granular branch that learns meso-scale features of the image, including shapes and object edges in the image;

[0020] A coarse-grained branch learns coarse-scale features of the image, wherein the mesoscale features include composition and spatial relations in the image.

[0021] According to the method of the first embodiment of the present invention, the fine-grained branch is: fine (I; θ fine )=DCN(I), where F fine represents the fine-grained branch, DCN represents the deep convolutional network, I is the input image, θ fine is the set of parameters of the deep convolutional network.

[0022] According to the method of the first embodiment of the present invention, the medium-sized branch is: F medium (I; θ medium )=EdgeDetection(Convolution(I)), where F medium represents the medium-granularity branch, Convolution (I) represents multiple convolution layer operations for shape recognition, EdgegDetection represents the edge detection algorithm, θ medium is a parameter set of the medium-grained branch.

[0023] According to the method of the first embodiment of the present invention, the coarse-grained branch is: coarse (I; θ coarse )=STN(I), where F coarse represents the coarse-grained branch, STM represents the spatial transformation network, θ coarse Represents the parameter set of STN.

[0024] According to the method of the first embodiment of the present invention, determining the fusion weights of the features of the multiple scales by using the attention mechanism includes:

[0025] Use the shared fully connected layer to transform F fine ,F medium and F coarse Converted to the same dimension, the formula is expressed as: F′ fine =FC(F fine θ fc ),F′ medium =FC(F medium θ fc ),F′ coarse =FC(F coarse θ fc ), where F′ fine F after dimension conversion fine , F′ medium F after dimension conversion medium , F′ coarse F after dimension conversion coarse , FC represents the fully connected layer, θ fc Represents the parameters of the fully connected layer;

[0026] The fusion weight of the features of each scale is calculated through the attention mechanism, and its formula is expressed as: fine ,w medium ,w coarse =Softmax(FC(concat(F′ fine ,F′ medium ,F′ coarse );θ attn ), where w fine F′ fine The fusion weight, w medium F′ medium The fusion weight, w coarse F′ coarse The fusion weight of concat means connecting multiple features into a long vector, θ attn Represents the parameters of the fully connected layer, and Softmax is the normalization function.

[0027] According to the method of the first embodiment of the present invention, the fused image feature is: fusion =w fine ·F′ fine +w medium ·F′ medium +w coarse ·F′ coarse , where F fusion represents the fused image feature.

[0028] According to the method of the first embodiment of the present invention, the using the image evaluation model to learn the features of the interaction data includes: using a natural language processing module to extract the emotion and semantic information in the interaction data to obtain the features of the interaction data: user (U;θ user )=NLP(U), where F user represents the characteristics of the interaction data, U represents the interaction data, θ user is a set of parameters of the natural language processing module.

[0029] According to the method of the first embodiment of the present invention, determining and outputting the score of the image based on the fused image features and the features of the interactive data includes:

[0030] The fused image features are fused with the features of the interaction data using a multimodal fusion network to obtain a multimodal fusion feature: F final (F fusion ,F user θ final )=MultimodalFusion(F fusion ,F user ), where F final represents the multimodal fusion feature, MultimodalFusion represents the multimodal fusion network, θ final is a set of parameters of the multimodal fusion network;

[0031] Based on the multimodal fusion features, a score of the image is determined and output.

[0032] A second embodiment of the present invention discloses an image generation method based on image evaluation, which is used in an electronic device. The method comprises:

[0033] Step A, receiving a user's image generation request;

[0034] Step B, generating an image based on the image generation request;

[0035] Step C, evaluating the image using the image evaluation method according to any one of claims 1 to 10 to obtain a score for the image;

[0036] Step D, determining whether the score reaches a predetermined threshold; if it reaches the predetermined threshold, sending the image to the user; if it does not reach the predetermined threshold, repeating steps B to D.

[0037] A third embodiment of the present invention discloses an electronic device, which includes a memory storing computer executable instructions and a processor. When the instructions are executed by the processor, the electronic device implements the image evaluation method according to the first embodiment of the present invention.

[0038] A fourth embodiment of the present invention discloses an electronic device, which includes a memory storing computer-executable instructions and a processor. When the instructions are executed by the processor, the electronic device implements the image generation method based on image evaluation according to the second embodiment of the present invention.

[0039] A fifth embodiment of the present invention discloses a computer storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer is caused to execute the image evaluation method according to the first embodiment of the present invention.

[0040] A sixth embodiment of the present invention discloses a computer storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer is caused to execute the image generation method based on image evaluation according to the second embodiment of the present invention.

[0041] A seventh embodiment of the present invention discloses a computer program product, comprising computer executable instructions, wherein the instructions are executed by a processor to implement the image evaluation method according to the first embodiment of the present invention.

[0042] An eighth embodiment of the present invention discloses a computer program product, comprising computer executable instructions, wherein the instructions are executed by a processor to implement the image generation method based on image evaluation according to the second embodiment of the present invention.

[0043] Compared with the prior art, the main differences and effects of the embodiments of the present invention are:

[0044] In the present invention, through the adaptive feature fusion mechanism (attention mechanism), the image evaluation model can automatically adjust the contribution ratio of features from different scales in the final evaluation according to the image content and aesthetic tasks. This attention-based fusion method not only makes the fusion process more flexible, but also improves the model's adaptability to the aesthetic characteristics of different types of images. A pre-trained language model is used to analyze user interaction data such as the number of likes and comment content. In this way, the image evaluation model can extract the user's emotional and semantic information so that this information can be combined with image features in the future to provide a more personalized aesthetic evaluation. Based on the interaction and the user's historical interaction data, the parameters of the image evaluation model are adjusted so that the image evaluation model is fine-tuned according to the data of the service-side enterprise employees, emphasizing the learning of company value preferences. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of an image evaluation method according to an embodiment of the present application is shown.

[0046] Figure 2 A diagram showing an implementation scenario of an image evaluation method according to an embodiment of the present application.

[0047] Figure 3 A structural diagram of an image evaluation model according to an embodiment of the present application is shown.

[0048] Figure 4 A flowchart of an image generation method based on image evaluation according to an embodiment of the present application is shown.

[0049] Figure 5A An image generated by an image generation module according to an embodiment of the present application is shown.

[0050] Figure 5B Another image generated by the image generation module according to an embodiment of the present application is shown.

[0051] Figure 6 A hardware structure block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present application more clear, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0053] First, consider the limitations of existing technologies. For example, using models trained on the laion2B-en dataset and using CGANs to generate high-resolution realistic images from semantic label maps often fails to include image-text pairs of specific companies, resulting in the generated images being unable to meet the requirements of corporate brands and values. Also, although the method based on the SD open source model attempts to introduce embedding vectors of ambient light styles, it does not take into account indoor scenes (such as specific environments such as restaurants), which limits its practicality.

[0054] Existing aesthetic quality assessment methods evaluate the aesthetic quality of images based on semantic labels of images, which usually limits the size of input images, which inevitably loses a lot of important aesthetic information during the preprocessing process. In view of the above problems, an embodiment of the present application proposes an image evaluation method for electronic devices. Figure 1 A flow chart of the method is shown, the method comprising:

[0055] S102, receiving an image. Figure 2In the scenario shown, the user sends an image to the backend server 203 corresponding to the APP on the terminal (such as a smart phone) 201 via the network 202, and the image evaluation module 2031 in the backend server 203 receives the image. Alternatively, the user generates a request by inputting a picture in the APP on the terminal 201; then the request is sent to the image generation module 2032 of the backend server 203 via the network 202; the image generation module 2032 generates an image based on the request, and then the image evaluation module 2031 of the backend server 203 receives the generated image. The image evaluation module 2031 of the backend server 203 may have an image evaluation model. The aesthetic evaluation model network based on the image evaluation model is based on the pre-trained ResNet-v2 network, which has the functions of classifying and segmenting images at the same time, and can accurately generate and output M-layer aesthetic features of different levels, and uniformly output them into a feature map of a fixed size. Methods Use the image evaluation model:

[0056] S104, learning features of multiple scales of the image. For example, Figure 3 As shown, the image evaluation model 30 includes an image feature module 31. The image feature module 31 includes multiple branches for learning multiple scale features, and specifically may include a fine-grained branch 311 for learning fine-scale features of an image, a medium-grained branch 312 for learning medium-scale features of an image, and a coarse-grained branch 313 for learning coarse-scale features of an image. The fine-scale features here include texture and color distribution in the image, and the medium-scale features include composition and spatial relationships in the image.

[0057] The image evaluation module 2031 of the back-end server 203 sends the received image to the image evaluation model 30, and then the image feature module 31 of the image evaluation model 30 learns the features of the image, wherein the fine-grained branch 311 learns the fine-scale features of the image, the medium-grained branch 312 learns the medium-scale features of the image, and the coarse-grained branch 313 learns the coarse-scale features of the image.

[0058] S106, determine the fusion weights of features at multiple scales through an attention mechanism. For example, Figure 3 The image feature module 31 of the image assessment model 30 further includes an attention mechanism 314. The attention mechanism 314 determines the fusion weights of fine-scale features, medium-scale features, and coarse-scale features.

[0059] S108, based on the fusion weight, the features of multiple scales are fused to obtain fused image features. Figure 2 and Figure 3From the above, the fusion weight determined by the attention mechanism 314 fuses the fine-scale features output by the fine-grained branch 311, the medium-scale features output by the medium-grained branch 312, and the coarse-scale features output by the coarse-grained branch 313 to fuse the image features.

[0060] S110, based on the fused image features, determine and output the image score. For example, in the image evaluation module 2031 of the backend server 203, the image evaluation model 30 determines the image score based on the fused image features. Then the image evaluation module 2031 outputs the image score. If the image being evaluated comes directly from the user's terminal 201, the image score will be sent to the terminal 201 by the end server 203 via the network 202. If the image being evaluated comes from the image generation module 2032 of the backend server 203, the image score will be used to determine whether the generated image meets the requirements.

[0061] In this application, the image evaluation model is fine-tuned based on the data of the service-side enterprise employees, emphasizing the learning of company value preferences; a variety of conditional information is added during modeling, such as the total number of social platforms of company employees, corporate value experience, corporate aesthetics experience, etc. Therefore, the scoring criteria of the image evaluation model can be adjusted to evaluate the brand characteristics of the service-side enterprise, such as the restaurant environment and employee image of a fast food restaurant.

[0062] S112, receiving the user's interaction with the rating. For example, the rating is sent to the user's terminal 201, and the user can express the degree of recognition of the rating on the APP of the terminal 201 (i.e., interaction), for example, by selecting an option of recognition or non-recognition; the user's recognition degree is sent to the back-end server 203 via the network 202 for reception. For another example, the image generation module 2032 of the back-end server 203 generates an image, the image evaluation module 2031 scores the generated image, and after judging that the image meets the requirements based on the score, the image is sent to the user's terminal 201 via the network 202; the user can interact with the image on the APP of the terminal 201, including: rating the image, indicating whether to accept the image, and whether to request to regenerate the image; since the generated image is sent to the user after being rated, the interaction of these users with the generated image indirectly indicates the user's recognition degree of the rating.

[0063] S114, adjusting the parameters of the image evaluation model based on the interaction and the user's historical interaction data. The user's current interaction and historical interaction can be used as labels for the scores given to the image evaluation model 30, and the image evaluation model 30 can be trained using these labeled scores.

[0064] In the present application, the fine-scale features processed by the fine-grained branch include the texture and color distribution of the image. The fine-grained branch 311 of the image feature module 31 in the image evaluation model 30 is: fine (I; θ fine )=DCN(I), where F fine represents the fine-grained branch, DCN represents the deep convolutional network, I is the input image, θ fine It is a set of parameters of a deep convolutional network. The deep convolutional network learns the local features of the image through multiple layers of filters, and the complexity of the features captured by each layer gradually increases. In this way, the network can learn image features from simple to complex, which will be used for subsequent aesthetic evaluation.

[0065] In the present application, the mesoscale features processed by the mesoscale branch include the shape of the image and the edge of the object. The mesoscale branch 312 of the image feature module 31 in the image evaluation model 30 is: medium (I; θ medium )=EdgeDetection(Convolution(I)), where F medium represents the medium-granularity branch, Convolution (I) represents multiple convolutional layer operations for shape recognition, EdgeDetection represents the edge detection algorithm, θ medium is the parameter set of the medium-granularity branch. This branch first uses the edge detection algorithm to preliminarily extract the structural information of the image, and then further refines this information through the convolution layer to capture features such as shape and contour. They provide medium-level visual information for the aesthetic evaluation of the image.

[0066] In the present application, the coarse-scale features processed by the coarse-grained branch include the macroscopic layout of the image, such as composition and spatial relationship. The coarse-grained branch 313 of the image feature module 31 in the image evaluation model 30 is: coarse (I; θ coarse )=STN(I), where F coarse represents the coarse-grained branch; STN represents the spatial transformation network, which can transform the input image, learn the global transformation of the image, and adjust the perspective and scale of the image to capture the overall layout, composition, and composition of the image. coarse Represents the parameter set of STN.

[0067] In this application, the attention mechanism is used as an adaptive feature fusion mechanism, which can dynamically adjust the fusion weights of features of different scales according to the characteristics of the image content and the aesthetic task, so that the image evaluation model automatically determines the importance of fine-scale features, medium-scale features, and coarse-scale features in the final aesthetic evaluation. The attention mechanism 314 of the image feature module 31 in the image evaluation model 30 includes: using a shared fully connected layer to integrate F fine ,F medium and F coarse Converted to the same dimension, the formula is expressed as: F′ fine =FC(F fine θ fc ),F′ medium =FC(F medium θ fc ),F′ coarse =FC(F coarse θ fc ), where F′ fine F after dimension conversion fine , F′ medium F after dimension conversion medium , F′ coarse F after dimension conversion coarse , FC represents the fully connected layer, θ fc represents the parameters of the fully connected layer; the fusion weight of the features of each scale is calculated, and its formula is expressed as: fine ,w medium ,w coarse =Softmax(FC(concat(F′ fine ,F′ medium ,F′ coarse );θ attn ), where w fine F′ fine The fusion weight, w medium F′ medium The fusion weight, w coarse F′ coarse The fusion weight of concat means connecting multiple features into a long vector, θ attn Represents the parameters of the fully connected layer, and Softmax is the normalization function.

[0068] According to some embodiments of the present application, the fused image feature is: fusion =w fine ·F′ fine +w medium ·F′ medium +w coarse ·F′ coarse , where F fusionRepresents the fused image features.

[0069] In summary, the implementation of the feature fusion mechanism of this application can be expressed as an adaptive fusion function F:

[0070] F fusion =F(F fine ,F medium ,F coarse θ fusion )

[0071] Among them, θ fusion is a set of parameters of the adaptive fusion network. The network automatically adjusts the fusion weights of features of different scales through a training process to optimize the evaluation results. The network takes the features extracted from each branch as input and learns how to combine these features most effectively through training. The network outputs a weight vector indicating the importance of each feature in the final aesthetic evaluation.

[0072] Through the adaptive feature fusion mechanism (attention mechanism) of this application, the image evaluation model can automatically adjust the contribution of features from different scales in the final evaluation according to the image content and aesthetic tasks. This attention-based fusion method not only makes the fusion process more flexible, but also improves the model's adaptability to the aesthetic characteristics of different types of images.

[0073] According to some embodiments of the present application, the method further includes: receiving user interaction data, the interaction data including the number of likes and the content of comments; using the image evaluation model to learn the features of the interaction data, specifically, Figure 3 As shown, it includes: using the natural language processing module 32 of the image evaluation model 30 to extract the emotion and semantic information in the interaction data, and obtaining the characteristics of the interaction data: F user (U;θ user )=NLP(U), where F user represents the characteristics of the interaction data, U represents the interaction data, θ user It is a set of parameters for the natural language processing module. The NLP module uses a pre-trained language model to analyze user interaction data such as the number of likes and comment content. In this way, the image evaluation model can extract the user's emotional and semantic information so that this information can be combined with image features to provide a more personalized aesthetic evaluation.

[0074] Determining and outputting the image score based on the fused image features also includes: determining and outputting the image score based on the fused image features and the features of the interaction data. The image evaluation model of the present application not only relies on the features of the image itself, but also can combine user interaction data, such as the number of likes, comment content, etc., to enhance the accuracy and personalization of aesthetic evaluation.

[0075] According to some embodiments of the present application, based on the fused image features and the features of the interactive data, determining and outputting the score of the image includes: using a multimodal fusion network to fuse the fused image features with the features of the interactive data to obtain a multimodal fusion feature: F final (F fusion ,F user θ final )=MultimodalFusion(F fusion ,F user ), where F final Represents multimodal fusion features, MultimodalFusion represents multimodal fusion network, θ final It is a parameter set of the multimodal fusion network; based on the multimodal fusion features, the image score is determined and output. In this application, the method dynamically adjusts the image evaluation criteria by monitoring the user's interaction history, which can better adapt to the user's personalized needs. In other words, the image evaluation model can not only learn based on the direct feedback given by the user, but also extract preference information from the user's behavior pattern, thereby continuously optimizing its evaluation algorithm.

[0076] According to some embodiments of the present application, in order to dynamically learn user preferences and adjust evaluation criteria, the image evaluation model can continuously monitor the user's interaction history and update network parameters, namely:

[0077] For each user interaction:

[0078] Update based on user interaction data: θ fine ,θ medium ,θ coarse ,θ fusion ,θ user ,θ final

[0079] This recognition method can help image evaluation models capture subtle differences that traditional image processing methods may overlook, such as sociocultural trends, popular elements, and individual aesthetic differences.

[0080] The embodiment of the present application also proposes an image generation method based on image evaluation, which is used in electronic equipment. Figure 4 A flow chart of the method is shown, the method comprising:

[0081] S402, receiving a user's image generation request. Figure 2In the scenario shown, the user generates a request by inputting a picture in the APP of the terminal 201. The request may be a natural language statement, such as: "Generate a picture of me dining in a western fast food restaurant"; the request is then sent to the back-end server 203 via the network 202, and the image generation module 2032 of the back-end server 203 receives the request.

[0082] S404, based on the image generation request, generate an image. For example, the image generation module 2032 generates an image based on the user's request "generate a picture of me eating at a western-style fast food restaurant", such as Figure 5A shown.

[0083] S406: Evaluate the image using the image evaluation method according to the embodiment of the present application to obtain a score for the image. Figure 5A The generated image shown is evaluated and a score is output, for example, 70 points.

[0084] S408, determine whether the score reaches a predetermined threshold. For example, the score can be 0-100 points, and the predetermined threshold is 85 points. Figure 5A The generated image shown has a score of 70, which does not meet the predetermined threshold.

[0085] - If the predetermined threshold is reached, the image is sent to the user at S410;

[0086] - If the predetermined threshold is not reached, then S404-S408 are repeated. Figure 5A The score of the generated image shown does not reach the predetermined threshold, so S404-S408 are repeated. In the repeated S404, the image generation module 2032 generates a new image based on the user's request "generate a picture of me eating in a western-style fast food restaurant", and the image is as follows Figure 5B In the repeated S406, the image evaluation module 2031 of the back-end server 203 receives the image evaluation module 2031 of the back-end server 203. Figure 5B The generated image shown in FIG. 1 is evaluated and a score is output, for example, 90 points. In the repeated S408, it is determined that Figure 5B The newly generated image shown in FIG. 1 reaches the threshold, and then in S410 the image is Figure 5B The newly generated image shown is sent to the user's terminal 201 via the network 202 .

[0087] The implementation scheme in this application utilizes an aesthetic evaluation model to generate images that conform to corporate brand characteristics and improve the quality and applicability of image generation.

[0088] Figure 6 It is a hardware structure block diagram of an electronic device implementing an embodiment of the present application.

[0089] like Figure 6 As shown, the electronic device 600 may include one or more processors 602, a system motherboard 608 connected to at least one of the processors 602, a system memory 604 connected to the system motherboard 608, a non-volatile memory (NVM) 606 connected to the system motherboard 608, and a network interface 610 connected to the system motherboard 608.

[0090] The processor 602 may include one or more single-core or multi-core processors. The processor 602 may include any combination of a general-purpose processor and a dedicated processor (e.g., a graphics processor, an application processor, a baseband processor, etc.). In an embodiment of the present invention, the processor 602 may be configured to execute one or more embodiments according to various method embodiments of the present application.

[0091] In some embodiments, system board 608 may include any suitable interface controller to provide any suitable interface to at least one of processors 602 and / or any suitable device or component in communication with system board 608 .

[0092] In some embodiments, the system board 608 may include one or more memory controllers to provide an interface to the system memory 604. The system memory 604 may be used to load and store data and / or instructions. In some embodiments, the system memory 604 of the electronic device 600 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

[0093] NVM 606 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM 606 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (CompactDisc) drive, and a DVD (Digital Versatile Disc) drive.

[0094] NVM 606 may include a portion of storage resources installed on a device of electronic device 600, or it may be accessible by the device but not necessarily a portion of the device. For example, NVM 606 may be accessed over a network via network interface 610.

[0095] In particular, the system memory 604 and the NVM 606 may include, respectively, a temporary copy and a permanent copy of the instruction 620. The instruction 620 may include instructions that, when executed by at least one of the processors 602, cause the electronic device 600 to implement a method as in an embodiment of the present application. In some embodiments, the instruction 620, hardware, firmware, and / or software components thereof may be additionally / alternatively placed in the system motherboard 608, the network interface 610, and / or the processor 602.

[0096] The network interface 610 may include a transceiver for providing a radio interface for the electronic device 600, thereby communicating with any other suitable device (e.g., a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the network interface 610 may be integrated with other components of the electronic device 600. For example, the network interface 610 may be integrated with at least one of the processor 602, the system memory 604, the NVM 606, and a firmware device (not shown) having instructions, and when at least one of the processors 602 executes the instructions, the electronic device 600 implements one or more embodiments of the various method embodiments of the present application.

[0097] The network interface 610 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 610 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0098] In one embodiment, at least one of the processors 602 may be packaged together with one or more controllers for the system motherboard 608 to form a system in package (SiP). In one embodiment, at least one of the processors 602 may be integrated on the same die with one or more controllers for the system motherboard 608 to form a system on a chip (SoC).

[0099] The electronic device 600 may further include an input / output (I / O) device 612 connected to the system motherboard 608. The I / O device 612 may include a user interface to enable a user to interact with the electronic device 600; the design of the peripheral component interface enables the peripheral components to interact with the electronic device 600. In some embodiments, the electronic device 600 further includes a sensor for determining at least one of an environmental condition and location information related to the electronic device 600.

[0100] In some embodiments, I / O device 612 may include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.

[0101] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0102] In some embodiments, the sensors may include, but are not limited to, gyroscope sensors, accelerometers, proximity sensors, ambient light sensors, and positioning units. The positioning unit may also be part of or interact with the network interface 610 to communicate with components of a positioning network (e.g., global positioning system (GPS) satellites).

[0103] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 600. In other embodiments of the present application, the electronic device 600 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0104] Program code can be applied to input instructions to perform the functions described in the present invention and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a system for processing instructions including processor 602 includes any system having a processor such as a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0105] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in the present invention is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0106] One or more aspects of at least one embodiment may be implemented by instructions stored on a computer-readable storage medium, which, when read and executed by a processor, enables an electronic device to implement the method of the embodiment described in the present invention.

[0107] According to some embodiments of the present application, a computer storage medium is disclosed, on which instructions are stored. When the instructions are executed on a computer, the computer executes the image evaluation method and the image generation method based on image evaluation according to the embodiments of the present application.

[0108] The method embodiment of the present application corresponds to this embodiment, and this embodiment can be implemented in conjunction with the method embodiment of the present application. The relevant technical details mentioned in the method embodiment of the present application are still valid in the present embodiment, and are not repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the method embodiment of the present application.

[0109] According to some embodiments of the present application, a computer program product is disclosed, including computer executable instructions, which are executed by a processor to implement an image evaluation method and an image generation method based on image evaluation according to an embodiment of the present application.

[0110] The method embodiment of the present application corresponds to this embodiment, and this embodiment can be implemented in conjunction with the method embodiment of the present application. The relevant technical details mentioned in the method embodiment of the present application are still valid in the present embodiment, and are not repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the method embodiment of the present application.

[0111] It is to be understood that the specific embodiments described herein are only for explaining the present application, rather than limiting the present application. In addition, for ease of description, only some but not all structures or processes related to the present application are shown in the accompanying drawings. It should be noted that in this specification, similar reference numerals and letters represent similar items in the accompanying drawings.

[0112] It should be understood that although the terms "first", "second", etc. may be used herein to describe various features, these features should not be limited by these terms. These terms are used only to distinguish and should not be understood as indicating or implying relative importance. For example, a first feature may be referred to as a second feature, and similarly a second feature may be referred to as a first feature without departing from the scope of the exemplary embodiments.

[0113] In the description of this application, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this embodiment can be understood according to specific circumstances.

[0114] Illustrative embodiments of the present application include, but are not limited to, image evaluation and image generation methods, apparatus, media, and computer program products based on image evaluation.

[0115] The various aspects of the illustrative embodiments will be described using terms commonly used by those skilled in the art to convey the essence of their work to other persons skilled in the art. However, it is apparent to those skilled in the art that some alternative embodiments may be implemented using the features described in part. For purposes of explanation, specific numbers and configurations are set forth to provide a more thorough understanding of the illustrative embodiments. However, it is apparent to those skilled in the art that alternative embodiments may be implemented without the specific details. In some other cases, some well-known features are omitted or simplified herein to avoid obscuring the illustrative embodiments of the present application.

[0116] In addition, various operations will be described as multiple separate operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be interpreted as implying that these operations must rely on the order of description, and many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can also be rearranged. When the described operations are completed, the process can be terminated, but can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0117] References in the specification to "one embodiment," "an embodiment," "an illustrative embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or property, but each embodiment may or may not necessarily include the particular feature, structure, or property. Moreover, these phrases are not necessarily referring to the same embodiment. In addition, when particular features are described in conjunction with a specific embodiment, the knowledge of those skilled in the art can affect the combination of these features with other embodiments, whether or not these embodiments are explicitly described.

[0118] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A and / or B" means "(A), (B), or (A and B)."

[0119] As used herein, the term "module" may refer to, be part of, or include: memory (shared, dedicated, or group) for running one or more software or firmware programs, application-specific integrated circuits (ASICs), electronic circuits and / or processors (shared, dedicated, or group), combinational logic circuits, and / or other suitable components that provide the described functionality.

[0120] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order is not necessary. Instead, in some embodiments, these features may be described in a manner and / or order different from that shown in the illustrative drawings. In addition, the structural or method features included in a specific figure do not mean that all embodiments need to include such features. In some embodiments, these features may not be included or may be combined with other features.

[0121] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented in the form of instructions or programs carried or stored on one or more transient or non-transient machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors, etc. When the instructions or programs are run by the machine, the machine may perform the aforementioned various methods. For example, the instructions may be distributed via a network or other computer-readable medium. Therefore, the machine-readable medium may include, but is not limited to, any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, such as a floppy disk, an optical disk, a compact disk read-only memory (CD-ROMs), a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electronically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, or a flash memory or a tangible machine-readable memory for transmitting network information via electrical, optical, acoustic, or other forms of signals (e.g., carrier waves, infrared signals, digital signals, etc.). Accordingly, machine-readable media includes any form of machine-readable media suitable for storing or transmitting electronic instructions or information readable by a machine (eg, a computer).

[0122] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the use of the technical solution of the present application is not limited to the various applications mentioned in the embodiments of the present application. Various structures and modifications can be easily implemented with reference to the technical solution of the present application to achieve the various beneficial effects mentioned herein. Various changes made within the knowledge scope of ordinary technicians in the field without departing from the purpose of the present application should all fall within the scope of the patent coverage of the present application.

Claims

1. An image evaluation method for electronic equipment, characterized in that: The method comprises: receiving an image; Evaluate the model using images: Learning features of the image at multiple scales, The fusion weights of the features of the multiple scales are determined by the attention mechanism. The features of the multiple scales are fused based on the fusion weight to obtain a fused image feature, Based on the fused image features, determining and outputting a score of the image, receiving user interactions with the ratings; Based on the interaction and the user's historical interaction data, parameters of the image evaluation model are adjusted.

2. The method according to claim 1, characterized in that The method further includes: receiving interaction data of the user, the interaction data including the number of likes and the content of comments; and learning the features of the interaction data using the image evaluation model; The determining and outputting the score of the image based on the fused image features further includes: determining and outputting the score of the image based on the fused image features and features of the interaction data.

3. The method according to claim 2, characterized in that The image evaluation model includes an image feature module, and the image feature module includes: A fine-grained branch that learns fine-scale features of the image, including texture and color distribution in the image; A meso-granular branch that learns meso-scale features of the image, including shapes and object edges in the image; A coarse-grained branch learns coarse-scale features of the image, wherein the mesoscale features include composition and spatial relations in the image.

4. The method according to claim 3, characterized in that The fine-grained branches are: fine (I; θ fine )=DCN(I), where F fine represents the fine-grained branch, DCN represents the deep convolutional network, I is the input image, θ fine is the set of parameters of the deep convolutional network.

5. The method according to claim 4, characterized in that The medium-sized branch is: F medium (I; θ medium )=EdgeDetection(Convolution(I)), where F medium represents the medium-granularity branch, Convolution (I) represents multiple convolution layer operations for shape recognition, EdgeDetection represents the edge detection algorithm, θ medium is a parameter set of the medium-grained branch.

6. The method according to claim 5, characterized in that The coarse-grained branches are: coarse (I; θ coarse )=STN(I), where F coarse represents the coarse-grained branch, STN represents the spatial transformer network, θ coarse Represents the parameter set of STN.

7. The method according to claim 6, characterized in that The determining the fusion weights of the features of the multiple scales by the attention mechanism includes: Use the shared fully connected layer to transform F fine ,F medium and F coarse Converted to the same dimension, the formula is expressed as: F′ fine =FC(F fine θ fc ),F′ medium =FC(F medium θ fc ),F′ coarse =FC(F coarse θ fc ), where F′ fine F after dimension conversion fine , F′ medium F after dimension conversion medium , F′ coarse F after dimension conversion coarse , FC represents the fully connected layer, θ fc Represents the parameters of the fully connected layer; The fusion weight of the features of each scale is calculated through the attention mechanism, and its formula is expressed as: fine ,w medium ,w coarse =Softmax(FC(concat(F′ fine ,F′ medium ,F′ coarse );θ attn ), where w fine F′ fine The fusion weight, w medium F′ medium The fusion weight, w coarse F′ coarse The fusion weight of concat means connecting multiple features into a long vector, θ attn Represents the parameters of the fully connected layer, and Softmax is the normalization function.

8. The method according to claim 7, characterized in that The fused image feature is: fusion =w fine ·F′ fine +w medium ·F′ medium +w coarse ·F′ coarse , where F fusion represents the fused image feature.

9. The method according to claim 8, characterized in that The using the image evaluation model to learn the features of the interaction data includes: using a natural language processing module to extract sentiment and semantic information in the interaction data to obtain the features of the interaction data: user (U;θ user )=NLP(U), where F user represents the characteristics of the interaction data, U represents the interaction data, θ user is a set of parameters of the natural language processing module.

10. The method according to claim 9, characterized in that The determining and outputting the score of the image based on the fused image feature and the feature of the interactive data comprises: The fused image features are fused with the features of the interaction data using a multimodal fusion network to obtain a multimodal fusion feature: F final (F fusion ,F user θ final )=MultimodalFusion(F fusion ,F user ), where F final represents the multimodal fusion feature, MultimodalFusion represents the multimodal fusion network, θ final is a set of parameters of the multimodal fusion network; Based on the multimodal fusion features, a score of the image is determined and output.

11. An image generation method based on image evaluation, used in electronic equipment, characterized in that: The method comprises: Step A, receiving a user's image generation request; Step B, generating an image based on the image generation request; Step C, evaluating the image using the image evaluation method according to any one of claims 1 to 10 to obtain a score for the image; Step D, determining whether the score reaches a predetermined threshold; if it reaches the predetermined threshold, sending the image to the user; if it does not reach the predetermined threshold, repeating steps B to D.

12. An electronic device, characterized in that: The electronic device comprises a memory storing computer executable instructions and a processor. When the instructions are executed by the processor, the electronic device implements the image evaluation method according to any one of claims 1 to 10.

13. An electronic device, characterized in that: The electronic device includes a memory storing computer executable instructions and a processor. When the instructions are executed by the processor, the electronic device implements the image generation method based on image evaluation according to claim 11.

14. A computer storage medium, characterized in that: Instructions are stored on the computer storage medium. When the instructions are executed on a computer, the computer is enabled to execute the image evaluation method according to any one of claims 1 to 10.

15. A computer storage medium, characterized in that Instructions are stored on the computer storage medium. When the instructions are executed on a computer, the computer is caused to execute the image generation method based on image evaluation according to claim 11 .

16. A computer program product, characterized in that The method comprises computer executable instructions, wherein the instructions are executed by a processor to implement the image evaluation method according to any one of claims 1 to 10.

17. A computer program product, characterized in that The method comprises computer executable instructions, wherein the instructions are executed by a processor to implement the image generation method based on image evaluation according to claim 11.

Images