Method for sharing cloud picture

Abstract

The embodiment of the invention discloses a method for sharing cloud pictures. The method is used for improving the efficiency and expressive power of sharing the cloud pictures. The scheme is applied to a cloud server, and the method comprises the steps of obtaining picture feature information provided by a target user, the picture feature information being used for representing features possessed by a picture that the target user intends to share; performing feature matching on a plurality of pictures stored in the cloud by the target user based on the picture feature information to obtain at least one to-be-selected picture; providing at least one to-be-selected picture for the target user, and obtaining at least one target sharing picture selected by the target user in the at least one to-be-selected picture; building a copywriting prompt based on the picture feature information and / or the at least one target shared picture, and generating a shared copywriting through a large language model; and executing a content sharing operation on the at least one target sharing picture and sharing copywriting.

Description

Technical Field

[0001] The present application relates to the field of cloud image applications, and in particular to a method for sharing cloud images. Background Art

[0002] Based on the cloud space, Internet users can store photos and other images in the cloud, which not only saves the storage space of local devices, but also makes it easier for users to classify and manage images.

[0003] When users share pictures on the cloud, there is a cumbersome process. When sharing pictures, users need to select the pictures they want to share one by one. It is troublesome to select satisfactory pictures in scenes with a large number of pictures, which makes the sharing operation time-consuming. In addition, the content of the pictures shared is relatively simple, and users often need to edit the text themselves, which is weak in expression.

[0004] How to improve the efficiency and expressiveness of sharing cloud pictures is the technical problem to be solved by this application. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method for sharing cloud pictures, so as to improve the efficiency and expressiveness of sharing cloud pictures.

[0006] In a first aspect, a method for sharing cloud images is provided, which is applied to a cloud server and includes:

[0007] Acquire image feature information provided by a target user, wherein the image feature information includes feature text, and the feature text is used to indicate features of the image that the target user intends to share;

[0008] Perform feature matching on multiple pictures stored in the cloud by the target user based on the feature text to obtain at least one picture to be selected;

[0009] Providing the at least one picture to be selected to the target user, and acquiring at least one target shared picture selected by the target user from the at least one picture to be selected;

[0010] Building a copy prompt based on the picture feature information and / or the at least one target sharing picture, and generating a sharing copy through a large language model;

[0011] A content sharing operation is performed on the at least one target sharing picture and the sharing text.

[0012] In a second aspect, a device for sharing cloud pictures is provided, which is applied to a cloud server and includes:

[0013] A first acquisition module acquires picture feature information provided by a target user, wherein the picture feature information includes feature text, and the feature text is used to indicate features of the picture that the target user intends to share;

[0014] A matching module, performing feature matching on a plurality of images stored in the cloud by the target user based on the feature text, to obtain at least one image to be selected;

[0015] A second acquisition module provides the at least one to-be-selected picture to the target user, and acquires at least one target shared picture selected by the target user from the at least one to-be-selected picture;

[0016] A generation module, constructing a copy prompt based on the picture feature information and / or the at least one target sharing picture, and generating a sharing copy through a large language model;

[0017] The sharing module performs a content sharing operation on the at least one target sharing picture and the sharing text.

[0018] According to a third aspect, an electronic device is provided. The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the method according to the first aspect are implemented.

[0019] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method according to the first aspect are implemented.

[0020] In a fifth aspect, a computer program product is provided, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, the computer program being operable to cause a computer to execute part or all of the steps of the method of the first aspect.

[0021] In an embodiment of the present application, first, obtain the image feature information provided by the target user, and the image feature information is used to represent the features of the image that the target user intends to share; then, perform feature matching on multiple images stored in the cloud by the target user based on the feature text to obtain at least one candidate image; then, provide at least one candidate image to the target user, and obtain at least one target sharing image selected by the target user from at least one candidate image; then, construct a copy prompt based on the image feature information and / or at least one target sharing image, and generate a sharing copy through a large language model; finally, perform a content sharing operation on at least one target sharing image and sharing copy. Through this solution, it is possible to filter the candidate images that meet the user's sharing intention from multiple images stored in the cloud, and then after the target user selects the target sharing image, it is possible to automatically generate a sharing copy that matches the target sharing image, thereby realizing rapid image selection and automatic copy generation, which can effectively improve the efficiency of cloud image sharing, enrich the sharing content, and enhance the expressiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 This is one of the flowcharts of a method for sharing cloud pictures in one embodiment of the present application;

[0024] Figure 2a This is a second flow chart of a method for sharing cloud pictures in an embodiment of the present application;

[0025] Figure 2b This is a flowchart of a method for sharing cloud pictures in an embodiment of the present application;

[0026] Figure 2c This is a fourth flow chart of a method for sharing cloud pictures in an embodiment of the present application;

[0027] Figure 3a This is a flowchart of a method for sharing cloud pictures in an embodiment of the present application;

[0028] Figure 3b This is a sixth flow chart of a method for sharing cloud pictures in an embodiment of the present application;

[0029] Figure 4 This is a flowchart of a method for sharing cloud pictures in an embodiment of the present application, No. 7;

[0030] Figure 5This is a flowchart of a method for sharing cloud pictures in an embodiment of the present application;

[0031] Figure 6a This is a ninth flowchart of a method for sharing cloud pictures in an embodiment of the present application;

[0032] Figure 6b This is a flowchart of a method for sharing cloud pictures in one embodiment of the present application;

[0033] Figure 6c This is a flowchart of a method for sharing cloud pictures in an embodiment of the present application;

[0034] Figure 7a This is a flowchart diagram twelfth of a method for sharing cloud pictures in an embodiment of the present application;

[0035] Figure 7b It is a structural schematic diagram of a picture description generation model of a method for sharing cloud pictures in an embodiment of the present application;

[0036] Figure 8 This is a flowchart of a method for sharing cloud pictures in an embodiment of the present application.

[0037] Figure 9a This is a flowchart diagram of a method for sharing cloud pictures in an embodiment of the present application.

[0038] Figure 9b This is a flowchart of a method for sharing cloud pictures in an embodiment of the present application;

[0039] Fig.10 This is a flowchart diagram of a method for sharing cloud pictures in an embodiment of the present application.

[0040] Fig.11 It is a structural diagram of a device for sharing cloud pictures according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. The numbering of the drawings in this application is only used to distinguish the various steps in the scheme, and is not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0042] The solution provided in the embodiment of the present application relates to the field of cloud image applications, specifically to the fields of image processing, artificial intelligence, computer vision, cloud technology and content generation. The solution provided in the embodiment of the present application can be applied to the application scenario of cloud photo sharing. Specifically, it can be used to share pictures in cloud storage space efficiently and with high quality. For example, share to designated network users, network social platforms, network social groups, etc.

[0043] In the related art, users can share pictures through multiple operation steps. For example, the user first selects several pictures that he intends to share in the cloud picture collection. Then, write a suitable copy sentence for the picture. Then select the target of the intended sharing, such as friends, social platforms, etc. Finally, share the selected pictures and copy. In this process of sharing pictures through multiple operation steps, users are required to perform operations multiple times, and the sharing process is cumbersome. Among them, users are required to select shareable photos one by one. In scenes with a large number of photos, it is more troublesome to select satisfactory photos, which makes the sharing operation time-consuming. Moreover, the sharing content of this method is single, and users are required to edit the sharing copy by themselves, which lacks the expressiveness of the photo scene. In addition, users are required to select the sharing target. The above multiple operations rely on manual execution by users, and there may be omissions and errors.

[0044] In order to solve the problems existing in the related art, the embodiment of the present application provides a method for sharing cloud pictures, which is applied to a cloud server, such as Figure 1 As shown, this scheme includes the following steps:

[0045] S11: Acquire picture feature information provided by a target user, where the picture feature information is used to indicate features of the picture that the target user intends to share.

[0046] The image feature information may be text input by the target user through the user terminal, uploaded images, or other forms of information. The user terminal may send the target user's image feature information to the cloud server. For example, the target user may describe the intended shared image in the form of keywords, phrases, sentences, or paragraphs. This may include the time information and location information corresponding to the intended shared image, as well as feature information such as the name and color of the object in the image.

[0047] The above text information can be used to search for matching images in the cloud image collection. Alternatively, the target user can upload a similar image to the image that he intends to share, for example, the target user uploads a photo of the current object, such as a photo of a plant or an animal. The photo can be used to search for similar images in the cloud image collection, thereby filtering out previous photos of the same object, and efficiently sharing the growth process of plants and animals.

[0048] S12: Perform feature matching on multiple pictures of the target user stored in the cloud based on the picture feature information to obtain at least one picture to be selected.

[0049] In this step, feature extraction is performed on the image feature information provided by the target user, and feature matching is performed on multiple images stored in the cloud by the target user based on the extracted features. Feature extraction is performed for the specific form of the image feature information, such as text feature extraction for text and image feature extraction for images.

[0050] Optionally, matching is performed in order of the computing power required for feature matching from small to large. For example, if the image feature information contains clear keyword information that can be used for screening, such as time information and location information, screening is performed first based on the time information and location information to significantly reduce the number of images that need to perform further feature matching.

[0051] Then, further matching is performed based on other feature information besides time and location information. This can effectively reduce the overall computing power required for feature matching and improve the efficiency of image screening.

[0052] Optionally, if too many pictures are matched, pictures with a high degree of match can be further selected based on a preset number to avoid providing too many pictures to be selected to the user. The preset number can be flexibly set according to the actual needs of the target users.

[0053] S13: providing the target user with the at least one picture to be selected, and obtaining at least one target shared picture selected by the target user from the at least one picture to be selected.

[0054] In this step, a preview image that is easy to browse can be generated and presented to the target user. For example, based on the user terminal of the target user, multiple preview images with slightly lower resolution than the original image are generated and sent to the user terminal in the form of an ordered group of images, so that the target user can quickly view the selected images and then quickly select the target sharing image that he intends to share.

[0055] After the target user selects the target shared picture on the user terminal, the user terminal can feed back the identifier of the location of the target shared picture in the ordered group map to the cloud server, and the cloud server can determine the target shared picture selected by the target user from the ordered group map according to the identifier.

[0056] By previewing pictures and picture labels, the amount of data exchanged between user terminals and cloud servers can be reduced, network resource consumption can be reduced, and overall efficiency can be improved.

[0057] S14: Constructing a copy prompt based on the picture feature information and / or the at least one target sharing picture, and generating a sharing copy through a large language model.

[0058] In this step, a sharing copy is generated through a large language model. Among them, a copy prompt Prompt is constructed with picture feature information and / or at least one target sharing picture, and then the copy prompt Prompt is input into a large language model (LLM) to generate a sharing copy according to the output result of the LLM. Since the picture feature information is provided by the target user, it can express which pictures the target user intends to share, and the target sharing picture is the picture selected by the target user for sharing, so the picture feature information and the target sharing picture can effectively characterize the characteristics of the target user's intention to share the picture. In this step, a copy prompt is constructed based on picture feature information and / or at least one target sharing picture, so that the constructed copy prompt can effectively express the characteristics of the target user's intention to share the picture, thereby generating a sharing copy that matches the target user's sharing intention through a large language model. In actual application scenarios, Prompt can be generated based on a preset template. For example, one or more keywords are first generated based on picture feature information and / or at least one target sharing picture, and then the keywords are filled into the preset template sentence. Among them, the preset template sentences are, for example, "Please return a text based on keyword A, keyword B, keyword C...", "Please return a text that expresses emotions based on keyword A, keyword B, keyword C...", etc.

[0059] S15: Performing a content sharing operation on the at least one target sharing picture and the sharing text.

[0060] In this step, the sharing operation can be directly performed on the above-mentioned target shared pictures and shared texts, and the specific implementation method of the sharing operation is related to the sharing target. For example, if you want to share to a friend, send the above-mentioned target shared pictures and shared texts to the friend. If you want to share to a social platform, the above-mentioned target shared pictures and shared texts are performed content sharing based on the sharing rules of the social platform. Optionally, before performing the content sharing operation, a preview of the sharing results can be provided to the target user. The target user can adjust the content of the shared text, the position order of multiple target shared pictures, etc. according to actual needs. Optionally, before performing the content sharing operation, beautification suggestions can be provided to the target user. The beautification suggestions include, for example, suggestions such as picture beautification and text beautification. Among them, picture beautification can include picture style templates. If the target user selects a beautification template, the beautification template can be applied to the target shared picture to optimize the display effect of the target shared picture. If the target user selects text beautification, matching symbol expressions can be added to the adjacent positions of the nouns and modal particles in the shared text to optimize the display effect of the shared text.

[0061] In an embodiment of the present application, first, image feature information provided by a target user is obtained, and the image feature information is used to represent the features of the image that the target user intends to share; then, feature matching is performed on multiple images stored by the target user in the cloud based on the feature text to obtain at least one candidate image; then, at least one candidate image is provided to the target user, and at least one target sharing image selected by the target user from the at least one candidate image is obtained; then, a text prompt is constructed based on the image feature information and / or at least one target sharing image, and a sharing text is generated through a large language model; finally, a content sharing operation is performed on at least one target sharing image and the sharing text.

[0062] Through this solution, it is possible to screen out candidate pictures that meet the user's sharing intention from multiple pictures stored in the cloud. After the target user selects the target sharing picture, it can also automatically generate sharing text that matches the target sharing picture, thereby realizing rapid picture selection and automatic text generation, which can effectively improve the efficiency of cloud picture sharing, enrich the sharing content, and enhance the expressiveness.

[0063] The embodiment of the present application can be applied to a cloud disk server to interact with the target user through various forms such as application programs or applets. The solution can flexibly and efficiently provide the target user with a picture and text sharing function. Furthermore, it can also be applied to generate albums such as selected good pictures, recommended stories, and memories of moments, which can help provide the target user with an emotional and story-telling collection of memories.

[0064] Next, this solution is further explained from the user side.

[0065] First, users can enter the cloud sharing portal through the program, interactively input the characteristic text of the intended shared picture, and describe the key information they want to share. Among them, the cloud sharing portal can be presented in the form of a virtual button, set in the menu bar of the cloud album or in the sharing interface of the cloud album, to facilitate user interaction. When the user clicks the virtual button of the cloud sharing portal, an independent dialog box can be pulled up to provide the user with an interface for typing text. Among them, the characteristic text typed by the user can be in various forms such as keywords, sentences or paragraphs.

[0066] Then, the cloud server can parse the key information input by the user through the smart sharing component, and complete the photo selection in the user's photo collection based on the key information to obtain at least one picture to be selected. Among them, the smart sharing component can parse the text used for typing to obtain the user's intention. This may include location, time, emotion and attribute features, etc. These features can be used to assist in image screening. After obtaining the user intention analysis results, the photo list can be filtered based on the user's intention to construct a photo subset containing the pictures to be selected.

[0067] Next, the selected subset of photos is presented to the user in the form of a list. The user can perform a secondary screening of the photo list to retain the photos that are ultimately desired to be shared, i.e., the target shared pictures. After the user confirms the photos to be shared, the intelligent sharing component of the cloud server can automatically generate sharing texts based on the photo theme and content, and support users to perform secondary editing of the texts and overlay the photo texts to generate effects. After the photos and texts to be shared are confirmed, the intelligent sharing component of the cloud server can filter and recommend sharing recipients based on the user relationship group of the target user for the user to choose, and give the user prompts for specific relationships.

[0068] Through the solution provided by the embodiment of the present application, users can simply express the pictures they intend to share through text descriptions, and the cloud server provides users with pictures to choose from through feature matching, which significantly simplifies the user's picture selection operation. In addition, it can also generate text that matches the pictures that the user intends to share, improve the overall expressiveness of the shared content, and achieve efficient and high-quality cloud picture sharing.

[0069] Based on the solution provided in the above embodiment, optionally, the image feature information includes feature text. The feature text can be in the form of keywords, phrases, sentences or paragraphs. Figure 2a As shown, in the above step S12, based on the picture feature information, feature matching is performed on multiple pictures stored in the cloud by the target user to obtain at least one picture to be selected, including the following steps:

[0070] S21: Perform natural language processing on the feature text to extract the central word in the feature text.

[0071] In this solution, the image feature information can be first analyzed to obtain the feature text. Then, natural language processing (NLP) is performed on the feature text to obtain the NLP analysis result, which is the central word of the feature text. The central word can specifically refer to a word that appears frequently in the semantics expressed by the feature text. If the feature text contains only a single word, the word can be directly determined as the central word. If the feature text contains multiple words, the central word can be identified based on the word's part of speech and the relationship between the words.

[0072] S22: Generate a prompt text containing the central word based on a preset intention analysis template, and input it into the intention analysis large language model to obtain the intention analysis result of the target user.

[0073] In this step, based on the preset intention analysis template, the above-mentioned core word is used to construct a prompt text Prompt, which is used to input into the intention analysis large language model to obtain the intention analysis result of the target user. The intention analysis result can represent the core intention of the semantic expression of the feature text provided by the user.

[0074] The above-mentioned preset intention analysis template can be pre-set according to actual needs. For example, the preset intention analysis template can be "Please list the symbols and activities related to {central word}", so as to substitute the central word extracted in the above steps into the above-mentioned intention analysis template to obtain the prompt text for input into the intention analysis large language model.

[0075] The following table shows the characteristic text entered by the user, the NLP analysis results, the synthesized prompt text, and the target user's intention analysis results:

[0076]

[0077]

[0078] Optionally, on the basis of determining the central word from the feature text, modifying words such as the adjectives "happy" and "unforgettable" may be further determined from the feature text based on part of speech and semantics.

[0079] These modifying words can record the emotions that the target user intends to express, and can help improve the accuracy of subsequent matching of candidate images, so that the selected candidate images are more in line with the emotions that the target user intends to express.

[0080] Optionally, if the feature text contains specific information such as time information, location information, etc. that can be used to efficiently filter images, it can be extracted in the NLP analysis step.

[0081] In the subsequent step of screening images, preliminary screening can be performed based on this specific information to significantly reduce the number of images that require feature matching, thereby reducing overall computing power consumption and improving screening efficiency.

[0082] S23: Screening multiple pictures stored in the cloud by the target user to obtain at least one picture to be selected that matches the intention analysis result.

[0083] In this step, image screening and matching is performed based on the intention analysis results of the above steps, so as to select at least one candidate image that meets the user's sharing intention from the images stored by the target user in the cloud.

[0084] In practical applications, the efficiency of filtering images can be improved by using hierarchical filtering. Figure 2bAfter analyzing the text input by the user through NLP, multiple categories of vocabulary may be obtained.

[0085] If the user input text contains clear intended time and intended location, the above-mentioned clear time information and location information can be extracted, and in the step of performing the screening, the time information and / or location information is first used to perform a preliminary screening.

[0086] This clear information can quickly and effectively exclude a large number of unqualified images from the target user's massive cloud images, retaining a smaller number of images. This can save a lot of computing power in the subsequent center word matching step and improve the efficiency of matching the selected images.

[0087] If the user input text contains adjectives, the emotion expressed by the adjectives can be analyzed, and pictures can be filtered based on the emotion expression, so that the matched selected pictures are more in line with the emotion that the target user intends to express.

[0088] For the central word extracted from the user input text, a prompt text Prompt containing the central word is generated by combining the preset intent analysis template, and then the intent analysis result is obtained by the intent analysis large language model, and the intent analysis result can include multiple different attribute features. In the step of performing image screening and matching, each attribute feature in the above intent analysis result can be used to perform feature screening and matching, so as to determine at least one image to be selected.

[0089] For example, Figure 2c As shown, when the feature text input by the user contains the location and time, for the photo collection stored in the cloud by the target user, the intended location can be used to perform location filtering first to obtain pictures that match the location filtering. Then, the pictures that match the location filtering are filtered using the intended time, so as to obtain a subset of photos that match the location information and time information. Furthermore, the intention attribute features obtained based on the central word through the large language model are used to perform feature matching filtering through the photo filtering model, so as to obtain a filtered photo collection. The pictures in the filterable photo collection are the determined pictures to be selected.

[0090] Through the solution provided in the embodiment of the present application, NLP analysis is performed based on the feature text input by the user to determine the central word, and the user's intention analysis result is determined through the intention analysis large language model, so as to filter at least one candidate image that matches the intention analysis result. The filtered candidate images can be made to fit the actual sharing intention of the target user, reducing the tedious operations of the target user in filtering images and improving the speed and quality of sharing images.

[0091] Based on the solution provided in the above embodiment, optionally, the intention analysis result includes multiple intention feature words and phrases. Among them, the intention feature words and phrases specifically refer to words or short sentences that express the intention of the target user. Among them, Figure 3a As shown, in the above step S23, screening is performed on multiple pictures stored in the cloud by the target user to obtain at least one picture to be selected that matches the intention analysis result, including the following steps:

[0092] S31: Perform feature slicing on the multiple pictures to be screened stored in the cloud by the target user and input the comparative language image pre-trained CLIP picture encoder to obtain the picture coding features of the multiple pictures to be screened, wherein the picture coding features include a three-dimensional matrix, and the three-dimensional matrix includes the number dimension of the pictures to be screened, the number dimension of the sub-picture blocks obtained by feature slicing, and the feature dimension of the sub-picture blocks.

[0093] Optionally, if the feature text provided by the target user contains time and place, a preliminary screening is performed based on the time and place, and this step is performed with the image after the preliminary screening. If the feature text does not contain information such as time and place, this step can be performed directly.

[0094] In this step, feature slicing is performed on any image to be screened, and it is divided into P non-overlapping sub-blocks. i Input the Contrastive Language–Image Pre-training (CLIP) visual encoder to obtain the image feature embedding of dimension V, which is represented by I i .

[0095] Assuming that the number of images to be screened is M, and each image can obtain P feature embeddings, then M photos can obtain M×P photo feature embeddings of dimension V, which can be represented by a three-dimensional real number matrix, which is represented by I∈R M×P×V , where R represents the set of real numbers.

[0096] S32: Input the intention analysis result into the CLIP text encoder to obtain text encoding features of the intention analysis result, wherein the text encoding features include a two-dimensional matrix, and the two-dimensional matrix includes the quantity dimension of the intention feature words and sentences and the feature dimension of the intention feature words and sentences.

[0097] In this solution, the intent analysis result is represented as F. Assume that the intent analysis result F contains N intent feature words and sentences. In this step, the intent analysis result F is input into the text encoder. Each sentence in the intent analysis result F is input into the text encoder for processing to obtain an attribute embedding of dimension L. The attribute embedding is represented as T, which can be represented as a matrix with N rows and L columns. Among them, each row of data in the matrix is ​​the feature embedding corresponding to an attribute. The matrix composed of all attribute feature embeddings can be represented as T∈R N×L , where R represents the set of real numbers.

[0098] S33: Perform similarity matching on the text encoding features and the image encoding features through a linear semantic mapping large language model to obtain at least one candidate image that matches the intent analysis result.

[0099] See also Figure 3b In this scheme, the CLIP image encoder is used to extract the image feature embedding I, and the CLIP text encoder is used to extract the attribute feature embedding T. Then, the linear semantic mapping large language model is used to perform similarity matching on the above text encoding features and image encoding features, and the matching score is calculated to quantify the degree of matching between the image to be screened and the target user's intention in numerical form.

[0100] Through the solution provided in the embodiment of the present application, the image features and text features are respectively extracted by a feature encoder, and then the images are screened by feature similarity matching, which can ensure that the selected images meet the user's sharing intentions, thereby reducing the tediousness of the user's sharing operation and improving the efficiency and quality of image sharing.

[0101] Based on the solution provided in the above embodiment, optionally, Figure 4 As shown, in the above step S33, similarity matching is performed on the text encoding features and the picture encoding features through a linear semantic mapping large language model to obtain at least one candidate picture matching the intention analysis result, including the following steps:

[0102] S41: Mapping the image encoding feature to a semantic space matching the text encoding feature through a linear semantic mapping large language model to obtain a mapping feature.

[0103] In practical applications, there is always a certain gap between the visual features and text features encoded by the model, and the solution provided in the embodiment of the present application uses a linear semantic mapping layer (LML). The linear semantic mapping layer parameter can be W, and the dimension is V×L.

[0104] In this step, the image feature I extracted by the above CLIP image encoder is mapped to a semantic space closer to the text attribute feature T, and the output mapping feature is I LML . See Figure 3b , where the dotted line indicates that the calculation path exists only in the model parameter learning stage. The mapping is shown in formula (1):

[0105]

[0106] S42: Perform a matching score on the mapping feature and the text encoding feature to obtain a matching score result, wherein the matching score result indicates the matching degree between each to-be-screened picture and the intention analysis result.

[0107] After feature mapping, the resulting mapped features can be closer to the semantic space of the text features, which is conducive to improving the effectiveness of feature matching. In this step, a matching score is performed on the mapped features and the text encoding features. The higher the matching score, the more the corresponding image meets the sharing intention expressed by the target user.

[0108] S43: Determine at least one to-be-selected picture that matches the intention analysis result from a plurality of to-be-screened pictures based on the matching score result.

[0109] Optionally, a certain number of to-be-screened pictures with the highest degree of similarity matching are determined as to-be-selected pictures, or, to-be-screened pictures with a degree of similarity matching greater than a preset degree are determined as to-be-selected pictures.

[0110] Through the solution provided in the embodiment of the present application, the image features are mapped to the text space, which can facilitate the matching and comparison of the image features and the text features, and improve the accuracy and effectiveness of the matching score.

[0111] Based on the solution provided in the above embodiment, optionally, Figure 5 As shown, in the above step S42, the mapping feature and the text encoding feature are scored for matching to obtain a matching scoring result, wherein the matching scoring result indicates the matching degree between each to-be-screened picture and the intention analysis result, respectively, and includes the following steps:

[0112] S51: Determine the similarity between the target image to be filtered and the target intent characteristic phrase according to the average value of the similarities between multiple sub-image blocks of the target image to be filtered among the multiple images to be filtered and the target intent characteristic phrases contained in the multiple intent analysis results.

[0113] In this step, for multiple images to be screened, the similarity D between each image to be screened and the text attribute feature is calculated. Assuming that the number of images to be screened is M, and the number of intent feature phrases in the intent analysis result is N, then the similarity D between the i-th image to be screened and the n-th intent feature phrase is calculated. in As shown in formula (2):

[0114]

[0115] Specifically, for any image to be screened, the similarity between all P image blocks of the image to be screened and the text feature of the nth intention feature phrase is averaged to obtain the similarity D in .

[0116] Furthermore, the similarity D between the i-th image to be screened and the N intention feature words in the intention analysis result is i The calculation can be expressed as formula (3):

[0117]

[0118] Among them, D i is a vector with a dimension of N, and the nth dimension data represents the similarity between the i-th image to be screened and the n-th intention feature phrase.

[0119] S52: construct a similarity result matrix according to the similarity between each image to be screened and each intention feature phrase.

[0120] In the above steps, it is necessary to calculate the similarity between the M images to be screened and the intended feature phrases. The above formulas (2) and (3) can be used to calculate the similarity between each image to be screened and the N intended feature phrases in turn. In this step, the similarity between the M images to be screened and the N intended feature phrases is calculated based on the multiple similarities determined in the above steps. The similarity result matrix is ​​expressed as D, as shown in formula (4):

[0121]

[0122] The similarity result matrix D is an M-row N-column matrix, where each row of data represents the similarity between a to-be-screened image and N intended feature words and sentences, such as D ij is the similarity between the i-th image to be screened and the j-th intent feature phrase.

[0123] S53: Determine the picture similarity values ​​corresponding to the multiple intention feature words and sentences based on the similarity result matrix, wherein the picture similarity value corresponding to the intention feature word and sentence is the average of the similarities of each similar picture in the similar picture group, and the similar picture group corresponding to the intention feature word and sentence includes a first preset number of similar pictures with the highest similarity to the intention feature word and sentence.

[0124] The similarity result matrix obtained in the above steps can represent the similarity between each image to be screened and the intended feature words and sentences. In this step, the matching score is further calculated based on the similarity represented by the similarity result matrix.

[0125] In this step, assuming that the first preset number is K, the influence of the deviation on the prediction result is reduced by calculating the Top K mean benchmark. Specifically, for any target intent feature phrase among multiple intent feature phrases, select the K images to be screened that have the highest similarity with the target intent feature phrase, and calculate the mean similarity between these K images to be screened and the target intent feature phrase, as shown in formula (5):

[0126]

[0127] Among them, the mean benchmark is represented by the symbol E, which is the above-mentioned image similarity value. E represents the benchmark deviation of the model for image recall based on the intent feature phrase. N intent feature phrases can obtain N benchmark deviations, so E is a vector with a dimension of N.

[0128] S54: Determine a matching score corresponding to each image to be screened based on the image similarity value and the similarity result matrix, wherein the matching score represents a comprehensive matching degree between the corresponding image to be screened and each intention feature phrase in the intention analysis result.

[0129] In this step, the matching score S between the image to be screened and the intent analysis result is calculated by combining the similarity result matrix and the mean benchmark E. For the matching score between the i-th image to be screened and all N intent feature phrases in the intent analysis result, the similarity between the i-th image to be screened and the N intent feature phrases minus the mean benchmark E is used, as shown in formula (6):

[0130] S=D i -E, i=1,2,....M Formula (6)

[0131] Here, i ranges from 1 to M. In this step, the matching scores between all M images to be screened and N intended feature words and sentences are calculated, and finally the matching score matrix S is obtained. S is a matrix with M rows and N columns. The i-th row of data S i Represents the matching score between the i-th image to be screened and N intent feature phrases.

[0132] Among them, if the value of an element in S is greater than 0, it indicates that the matching degree between the image to be screened corresponding to the element and the intent analysis result ranks above the mean. Conversely, if the value of an element in S is less than 0, it indicates that the matching degree between the image to be screened corresponding to the element and the intent analysis result ranks below the mean. This scheme can widen the gap between matching images and non-matching images, and can effectively realize image screening.

[0133] Furthermore, after obtaining the above S, in the above step S43, at least one to-be-selected picture matching the intention analysis result is determined from a plurality of to-be-screened pictures based on the matching score result, specifically including:

[0134] Based on S, a to-be-selected picture is determined from multiple to-be-selected pictures. Specifically, for each to-be-selected picture, if there is any intention feature phrase whose matching score with the to-be-selected picture meets the requirement, the to-be-selected picture is selected as the to-be-selected picture. On the contrary, if the matching scores of the to-be-selected picture and all the intention feature phrases do not meet the requirement, the to-be-selected picture is not selected as the to-be-selected picture, as shown in formula (7):

[0135]

[0136] Where C is an M-dimensional vector. Assume that the i-th position element C i If it is 1, it means that the i-th image to be filtered is selected as the image to be selected. On the contrary, if the element is 0, it means that the i-th image to be filtered is not selected.

[0137] Among them, σ is a constant threshold value, which can be flexibly set according to actual needs. For example, the optional value is [0, 1). When the matching score is greater than σ, the photo is considered to meet the requirements. The larger the constant threshold σ is, the stricter the matching requirements for the photo are, but it will also lead to fewer photo recalls. The smaller σ is, the looser the matching requirements for the photo are, and it is possible to match irrelevant photos. Therefore, σ needs to be selected and adjusted according to the actual scene.

[0138] Optionally, the pictures to be selected screened out by the solution provided by the above embodiment can be provided to the target user in the form of a list, so that the target user can select a target sharing picture from the list of pictures to be selected.

[0139] Optionally, when there are multiple pictures to be selected, the pictures can be sorted from high to low according to the matching score results of each picture to be selected, or sorted from high to low according to the picture quality, so as to display multiple pictures to be selected in an orderly manner to the target user. The picture quality includes but is not limited to the comprehensive score of multiple dimensions such as clarity, brightness, color vividness, content harmony, etc.

[0140] In actual applications, the target user can click to preview the selected pictures, and can also perform editing operations such as moving and checking, and flexibly implement operations such as changing the order of pictures, beautifying, replacing, or removing and adding pictures, until the target user confirms to submit the target sharing picture.

[0141] Through the solution provided in the embodiment of the present application, the similarity can be quantified in numerical form, and comprehensive similarity matching can be achieved based on the various intention feature words and phrases in the intention analysis results of the target user, thereby effectively screening out candidate pictures that meet the user's sharing intentions and improving the efficiency and quality of picture screening.

[0142] Based on the solution provided in the above embodiment, optionally, Figure 6a As shown, in the above step S14, constructing a copy prompt based on the picture feature information and / or the at least one target sharing picture, and generating a sharing copy through a large language model, includes the following steps:

[0143] S61: Inputting the at least one target shared picture into a picture description generation model to obtain a plurality of picture description sentences.

[0144] See also Figure 6b In this step, the target shared picture selected by the target user is input into the picture description generation model, and the picture description generation model generates a picture description sentence according to the characteristic content of the input target shared picture. There are multiple picture description sentences, which are used to perform scoring in the subsequent steps to select the better sentence for constructing the copy.

[0145] S62: Scoring the plurality of picture description sentences based on a first preset scoring rule to determine a second preset number of target picture description sentences.

[0146] In this step, the first preset scoring rule can be flexibly set according to actual needs, for example, including content richness, wording, character setting matching, etc. In this step, multiple picture description sentences are scored according to the first preset scoring rule to select the target picture description sentence. Among them, the second preset number can be pre-set according to actual needs.

[0147] S63: Constructing a text prompt based on the image feature information and / or the target image description sentence.

[0148] In this step, the text prompt is constructed based on the picture feature information and / or the target picture description sentence. Since the picture feature information is provided by the target user, it can express which pictures the target user intends to share. And the target picture description sentence is generated by the target sharing picture, and the target sharing picture is the picture that the target user actually selects to share. Therefore, the above picture feature information and target picture description sentence can effectively characterize the characteristics of the picture that the target user intends to share. In this step, the text prompt is constructed based on the picture feature information and / or the target picture description sentence, so that the constructed text prompt can effectively express the characteristics of the picture that the target user intends to share. See Figure 6c The user input text may specifically refer to the feature text in the picture feature information, the emotional expression may be the information that can express the user's emotion extracted by LLM based on the feature text, and the photo copy may be the target picture description sentence generated by the solution provided in any of the above embodiments.

[0149] In this step, a copy prompt is generated based on the combination of the above-mentioned multiple information, so that the generated copy prompt can fully express the target user's picture sharing intention and emotion, improve the degree of fit between the final generated sharing copy and the user's intention, and improve the quality of the final shared copy.

[0150] S64: Generate a large language model from the copy prompt input sentence copy to obtain multiple copy sentences to be selected.

[0151] In this step, the copy prompt Prompt input sentence copy is generated into a large language model to generate multiple candidate copy sentences that match the target user's sharing intention. Figure 6c The sentence copy generation large language model can specifically be LLMs (Large Language Models), which is used to output multiple candidate copy sentences according to the input copy prompt.

[0152] S65: Scoring the plurality of candidate copywriting sentences based on a second preset scoring rule to determine a third preset number of target copywriting sentences.

[0153] In this step, the second preset scoring rule can be flexibly set according to actual needs, for example, including content richness, wording, character setting matching, etc. In this step, the second preset scoring rule is used to score multiple candidate copywriting sentences to select the target copywriting sentence. Among them, the third preset number can be pre-set according to actual needs.

[0154] S66: Generate sharing copy based on the third preset number of target copy sentences.

[0155] In this step, the third preset number of target copy sentences can be merged to construct a copy paragraph containing multiple sentences, and the quality of the final shared copy can be improved by typesetting and beautification. Alternatively, one copy can be selected from the third preset number of target copy sentences as the final shared copy.

[0156] Through the scheme provided by the embodiment of the present application, a picture description generation model is used to generate N picture description sentences according to the target shared picture selected by the target user. Then, the N sentences are scored based on the first preset scoring rule, and the sentence with the highest score can be selected as the target picture description sentence. Then, by constructing a copy prompt, a large language model for copy generation is used to obtain multiple copy sentences to be selected, thereby obtaining the final sharing copy.

[0157] This solution can effectively improve the quality of sharing copy, ensure that the sharing copy matches the target user's sharing intention, and then matches the target sharing picture selected by the target user, thereby improving the overall quality of picture sharing.

[0158] Based on the solution provided in the above embodiment, optionally, the picture description generation model includes a visual feature processing module, a label attribute processing module, a matching attribute processing module and a description generation module;

[0159] like Figure 7a As shown, in the above step S61, the at least one target shared picture is input into the picture description generation model to obtain multiple picture description sentences, including the following steps:

[0160] S71: Perform visual encoding on the input image through the visual feature processing module, and perform self-attention calculation on the visual encoding result to obtain visual features.

[0161] In the embodiment of the present application, the picture description generation model structure is as follows Figure 7b As shown, it includes a visual feature processing module, a label attribute processing module, a matching attribute processing module and a description generation module.

[0162] S72: extracting label attribute data from the input image through the label attribute processing module, performing text encoding on the label attribute data to obtain image text features, and performing cross-attention processing on the visual features and the image text features to obtain image attention features.

[0163] S73: Performing text encoding on at least one attribute feature matched by the input image through the matching attribute processing module to obtain an attribute text feature, and performing self-attention calculation on the attribute text feature to obtain an attribute attention feature.

[0164] S74: The description generation module performs cross-attention processing on the attribute attention feature and the image attention feature connected based on the feature dimension to obtain a joint feature, and generates multiple image description sentences through a feedforward layer.

[0165] In the embodiment of the present application, the attribute feature may specifically include the intention feature phrases described in any of the above embodiments. The following is a further explanation of the present solution with reference to an example. For ease of description, x represents the input photo data, L represents the photo tag attribute data, and f represents the input photo data. IE , f TE , f F They represent the visual encoding module IE, the text encoding module TE and the fully connected layers (FC) module respectively. IE and d TE Represent the feature dimensions of IE and TE modules respectively, I SA is the result of self-attention calculation, and finally I AM In order to combine the attention features of the label attributes, by TE and I SA After performing cross-attention activation, it is obtained. In this solution, the visual encoding module IE in the image description generation model performs visual encoding on the input image, as shown in formula (8):

[0166] I CLIP = f IE (x) Formula (8)

[0167] The matching attribute processing module performs text encoding on at least one attribute feature matched by the input image through the text encoding module TE to obtain an attribute text feature, as shown in formula (9):

[0168] T TE =f TE(L) Formula (9)

[0169] The image description generation model performs self-attention calculation on the visual encoding result to obtain visual features, as shown in formula (10):

[0170]

[0171] The description generation module performs cross attention processing on the attribute attention feature and the image attention feature connected based on the feature dimension to obtain a joint feature, that is, the attention feature combined with the label attribute, by T TE and I SA After performing cross-attention, the joint feature is activated and obtained as shown in formula (11):

[0172]

[0173] The matching attribute feature processing in the picture description generation model is to add the attribute features that meet the matching conditions with the photo selected in the solution provided by the above embodiment to the photo description generation process, thereby enhancing the description generation effect. Figure 7b , the e attribute features corresponding to the input photo I are processed by the text encoding module TE and then processed by SA, T ESA It represents the output of the matching attribute feature e after being processed by the SA module, and the formula is shown in formula (12):

[0174]

[0175] Finally, the description generation module can also be called the description tokens generation module, which is used to connect the global visual features, label attribute attention features and matching attribute features to complete the description generation. Figure 7b , where Feed Forward is a two-layer linear mapping to enhance the expressiveness of joint features. The overall calculation formula is shown in equations (13) to (14):

[0176] I concat =[MLP(I CLIP ), I AM Formula (13)

[0177]

[0178] Among them, concat represents the feature dimension connection operation, MLP is a multi-layer perceptron structure, which can contain one or more fully connected layers, and W1, W2, b1, and b2 are the parameters of the number of Feed Forward layers. FF After that, the corresponding N description sentence tokens can be predicted and generated by the tokens generation model TP, and finally the generated description sentences are scored, and the sentence with the highest score is selected as the photo description sentence. Among them, the tokens generation model TP can be, for example, a GPT2, BERT model, etc.

[0179] Based on the solution provided in the above embodiment, optionally, Figure 8 As shown, in the above step S62, scoring the multiple picture description sentences based on the first preset scoring rule to determine a second preset number of target picture description sentences includes the following steps:

[0180] S81: Perform text encoding on a first picture description sentence to obtain a description sentence text feature, wherein the first picture description sentence is any one of the multiple picture description sentences.

[0181] In this step, text encoding is performed on any first picture description sentence in the picture description sentences to be scored to obtain a description tokens feature encoding for subsequent feature comparison.

[0182] Optionally, this step S81 can be executed in parallel with S82, or can also be executed after S82. The text features of the description statement are used to provide a data basis for performing feature comparison in the subsequent step S83.

[0183] S82: Determine a first matching degree score between the first picture description statement and the plurality of intention feature words and sentences.

[0184] In this step, the description sentence text feature can be expressed as P e描述 , which can be obtained by calculating the predicted probability by weighted description of attribute feature matching score, as shown in formula (15):

[0185]

[0186] For example, suppose the number of tokens in a description sentence is K, t k Output token for the Kth prediction, c k Represents t k The probability of prediction generation. S m is the similarity result matrix described in any of the above embodiments, and e is the attribute feature set corresponding to the input photo m. k Value based on t k and the attribute feature set e, if t k If it matches the nth attribute feature in e, then increase t k The scoring weight, w k The value is S nm , that is, w k The value is the matching score between the mth photo and the nth attribute feature. k If it does not match any of the attribute features in the matching attribute feature set e, then w k The value is 0.

[0187] S83: Determine a second matching degree score between the description sentence text feature and the image text feature.

[0188] In this step, the similarity between the feature encoding of the description tokens and the visual feature encoding of the image is calculated to express their matching degree, where f TEis the text encoding module described in any of the above embodiments. The matching degree between the description sentence and the picture content, i.e., the second matching degree score, can be expressed as P I描述 , as shown in the following formula (16):

[0189]

[0190] S84: Determine a sentence score of the first picture description sentence based on the first matching degree score and the second matching degree score.

[0191] In this step, the first matching degree score and the second matching degree score can be combined by summing or weighting to comprehensively express the matching degree, such as shown in formula (17):

[0192] P 描述 =P e描述 +P I描述 Formula (17)

[0193] S85: Determine a second preset number of picture description sentences with high sentence scores as target picture description sentences.

[0194] In this step, according to the arrangement of the above sentence scores from high to low, a second preset number of picture description sentences with high sentence scores are selected as the target picture description sentences. The second preset number can be flexibly set according to actual needs.

[0195] Through the solution provided in the embodiment of the present application, it is possible to realize the mth photo I m All the description sentences obtained are scored and the probability P is generated from the description sentences e描述 And the matching degree P between the description sentence and the picture content I描述 These two aspects achieve comprehensive scoring, making the scoring reasonable and effective, thereby improving the quality of the target image description sentence.

[0196] Based on the solution provided in the above embodiment, optionally, Figure 9a As shown, in the above step S63, constructing a copy prompt based on the picture feature information and / or the target picture description sentence includes the following steps:

[0197] S91: Generate a copywriting prompt based on the target image description sentence, the specified style type label and the image feature information.

[0198] In this step, a text prompt Prompt is generated based on the target image description sentence, the specified style type label and the image feature information. Among them, the target image description sentence can be obtained through the scheme provided by any of the above embodiments. The specified style type label can be a pre-set style label, such as "Tang poetry style", "Song poetry style", "prose style", "artistic conception style", etc. The image feature information is provided by the target user and is used to express the feature information of the user's intention to share the image. The text prompt generated by combining these three types of information can include the target user's sharing intention and specific style, so that the style can be integrated with the target user's intention in the subsequent steps to generate a text with a specific style.

[0199] For example, in this step, a copy prompt is generated, such as "Based on 'Pearl River Night View' [Photo Description], return 5 lines of Tang poetry style [Style Type] copy." Entering this copy prompt into LLM can obtain multiple copy contents as shown in the following table:

[0200]

[0201] The number of texts generated by LLM can be configured and adjusted, and the number of words in the returned text can also be limited, for example, not exceeding 15 words.

[0202] Optionally, the feature text, the emotion expressed, and the photo copy input by the target user can be used to jointly generate a copy prompt. The photo copy is the target picture description sentence generated by the solution provided in any of the above embodiments. The copy prompt is input into the LLM to obtain multiple copy sentences to be selected, as shown in the following table:

[0203]

[0204]

[0205] In the above step S65, scoring the plurality of candidate copywriting sentences based on the second preset scoring rule to determine a third preset number of target copywriting sentences includes the following steps:

[0206] S92: Perform weighted scoring on the multiple candidate text sentences based on at least one of a sentence comfort feature, a sentence fluency feature, and a style type feature to obtain a text sentence score for each candidate text sentence.

[0207] In this step, a weighted score is performed on multiple candidate copywriting sentences through at least one sentence feature of sentence comfort, sentence fluency, and style type, so that the copywriting sentence score can comprehensively reflect the overall quality of the corresponding candidate copywriting sentences. Among them, the score of the sentence comfort feature is used to evaluate whether the copywriting content contains offensive and offensive words. Specifically, the score can be implemented based on an offensive word dataset, such as Hateful Memes, and a comfort score model is obtained through training to perform sentence comfort feature scoring. In addition, you can also define and add an offensive word database for evaluation. 舒适 The score of sentence fluency is used to evaluate the coherence and readability of the content. Specifically, a T5-based language model Parrot model can be used to calculate the fluency score S 流畅 The score of the sentence style type feature is used to evaluate whether the generated copy is consistent with the required style type, which can be obtained by calculating the feature distance between the copy and the style vocabulary.

[0208] For example, in the step of determining the score of the sentence style type feature, a local benchmark word library can be set for each style. The word library has a size of thousands or hundreds, which can be determined according to the specific situation, as long as it contains classic words under the corresponding style. Assuming that the number of benchmark word library entries is V, feature extraction is performed on the benchmark word library and the generated copy, including sentence length, word frequency, grammatical structure features, etc. to obtain the word library feature P v and copywriting features P l Then the score calculation of the sentence style type feature is as shown in formula (18):

[0209]

[0210] Among them, assuming that the feature dimension is F, for each sentence, the copy feature P is generated l , calculate its feature P with all terms in the benchmark vocabulary v The style similarity score between them.

[0211] After obtaining the above three scores, the above three scores can be weighted by formula (19) to obtain the copywriting sentence score:

[0212] S 照片文案 =w1S 舒适 +w2S 流畅 +w3S 风格 Formula (19)

[0213] Among them, each weighting coefficient can be flexibly set according to actual needs to control the weight of each score. The score of the copywriting sentence can be determined by combining the above three aspects of the score.

[0214] S93: Determine a third preset number of candidate copywriting sentences with high copywriting sentence scores as target copywriting sentences.

[0215] In this step, the third preset number of candidate copywriting sentences with the highest copywriting sentence scores are selected as the copywriting content corresponding to the photo and returned to the target user. The third preset number can be flexibly set according to actual needs, for example, only one optimal copywriting is returned to the target user.

[0216] Through the solution provided in the embodiment of the present application, after the photo copy and theme copy are generated, they can be provided to the user for preview and editing, supporting editing and modification of the copy, and the user can choose whether to superimpose the photo copy on the photo content for sharing based on personal preferences.

[0217] Based on the solution provided in the above embodiment, optionally, Fig.10 As shown, in the above step S15, performing a content sharing operation on the at least one target sharing picture and the sharing copy includes the following steps:

[0218] S101: Acquire at least one associated user group of the target user, wherein the associated user group includes picture-associated users, and any picture-associated user has an associated relationship with the target user in the plurality of pictures stored in the cloud.

[0219] In the solution provided in the embodiment of the present application, the associated user group can be automatically obtained based on the social relationship of the target user, wherein the obtained information is the information authorized for use by the target user. Alternatively, at least one associated user group can also be determined based on the operation of the target user. For example, when a user chooses dynamic sharing, the intelligent sharing component of the cloud server can recommend sharing reception based on the sharing theme and user relationship group to facilitate user selection. For example, when the sharing theme is parent-child related, users in the "relative relationship" group are given priority, and when the sharing theme is work-related, users in the "colleague relationship" group are given priority. For example, the associated user group of the target user is sorted out based on user tag information, group information, and photo character relationships, and can be divided into four categories: {relatives, colleagues, friends, and people you may know}.

[0220] In practical applications, a user relationship mapping table can be pre-established to store the association relationship between the target user and the associated users. The association relationship can be obtained based on the image recognition of the target user stored in the cloud server. For example, in the target user's personal album, find out whether there are the same face photos, and mark the users who directly or indirectly have the same face photos as "people who may know each other". For example, if users A and B have the same face photos, and users B and C have the same face photos, then A, B, and C are marked as "people who may know each other".

[0221] The above three types of relationships, namely relatives, colleagues and friends, can be determined by group and user tag information.

[0222] For example, to be marked as a "relative", at least one of the following conditions must be met:

[0223] (1) Users A and B are in the same family group. (2) User A has tagged User B as a relative, such as father or daughter. (3) User A’s photos with the “wedding” or “parent-child” tags contain User B’s face, and User B is the main character.

[0224] The relationship is marked as "colleague" only if at least one of the following conditions is met: (1) User A and User B are in the same work group. (2) User A marks User B as a colleague, such as boss, leader, manager, etc.

[0225] The relationship tagging as “friend” must meet the following conditions: (1) User A tags user B, but the tagging does not fall into any of the relative or colleague categories, such as tagging the user’s name.

[0226] Optionally, after obtaining the above four types of relationship annotations, you can also integrate and remove duplicates by relationship priority. Among them, the relationship priority is "Relatives>Friends>Colleagues>People You May Know". If the final relationship group of user A is {"Relatives": B, "Colleagues": B, E, G, "Friends": C, E, "People You May Know": C, D, E, F}, then after deduplication and integration based on priority, you will get {"Relatives": B, "Colleagues": G, "Friends": C, E, "People You May Know": D, F}.

[0227] S102: Sharing the at least one target sharing picture and the sharing text with a target associated user group selected by the target user.

[0228] In this step, sharing is performed based on the target associated user group selected by the target user, and the shared content includes the above-mentioned target sharing pictures and sharing copy. Optionally, the sharing method can be determined according to the target associated user group selected by the target user, such as sending the above-mentioned shared content to the group of the target associated user group selected by the target user. Alternatively, the shared content is sent one by one to multiple associated users selected by the target user, etc.

[0229] Through the solution provided by the embodiment of the present application, intelligent photo sharing can be realized. Image screening and sharing recommendations can be realized based on the hard evidence information of the image simply provided by the user. The recommendation quality of images and texts is effectively improved by feature matching, thereby effectively improving the overall quality of shared content. Moreover, sharing recommendations can be realized based on the user's association relationship, effectively reducing the steps of manual operation by the user, improving the sharing efficiency of images and texts, and improving the overall quality of shared content.

[0230] In the solution provided by the embodiment of the present application, the sharing operation is intelligent and convenient. Users can automatically filter and generate text for shared photos by simply entering a line of text information, and can automatically prompt the recipient of the sharing based on the user relationship group, which improves the efficiency of dynamic photo sharing and saves user operation time. Moreover, sharing is more situational and diverse, and can automatically generate photo texts of various styles and themes based on photos, and consider the quality of texts from multiple dimensions and calculate the text scores, which can make the shared content more diverse and more in line with the emotions of the photos.

[0231] Compared with the related art, the solution provided by the embodiment of the present application solves the problem of dynamic sharing of cloud photos when the number of photos is huge and the user's literary expression ability is insufficient. At the same time, due to the advancement of data and service cloud work, data can be processed at low cost and high precision. For the majority of users, they can enjoy higher quality photo intelligent sharing services. For manufacturers, it reduces work costs and enhances user stickiness. In addition, the methods of photo and attribute matching, text generation and scoring, and user relationship group division proposed in this solution can also be promoted and applied to a variety of photo application scenarios in the future, such as photo selection recommendation system, album memoir generation system, photo subtitle generation, picture description and other interesting applications.

[0232] In order to solve the problems existing in the related art, the embodiment of the present application also provides a device 110 for sharing cloud pictures, which is applied to a cloud server, such as Fig.11 As shown, including:

[0233] A first acquisition module 111 acquires picture feature information provided by a target user, wherein the picture feature information includes feature text, and the feature text is used to indicate features of the picture that the target user intends to share;

[0234] A matching module 112 performs feature matching on multiple pictures stored in the cloud by the target user based on the feature text to obtain at least one picture to be selected;

[0235] The second acquisition module 113 provides the at least one selected picture to the target user, and acquires at least one target shared picture selected by the target user from the at least one selected picture;

[0236] A generating module 114 constructs a copy prompt based on the picture feature information and / or the at least one target sharing picture, and generates a sharing copy through a large language model;

[0237] The sharing module 115 performs a content sharing operation on the at least one target sharing picture and the sharing text.

[0238] Among them, the above modules in the device provided by the embodiment of the present application can also implement the method steps provided by the above method embodiment. Alternatively, the device provided by the embodiment of the present application can also include other modules in addition to the above modules to implement the method steps provided by the above method embodiment. And the device provided by the embodiment of the present application can achieve the technical effects that can be achieved by the above method embodiment.

[0239] Preferably, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, each process of the above-mentioned method for sharing cloud pictures is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor, each process of the above-mentioned method for sharing cloud pictures is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium is such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program can be operated to enable a computer to execute some or all of the steps of the above-mentioned method for sharing cloud pictures, and the same technical effect can be achieved. To avoid repetition, it is not repeated here.

[0240] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps of the functions specified in the blocks or multiple blocks. In a typical configuration, the computing device includes one or more processors (CPU), input / output interfaces, network interfaces and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium. Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be a computer-readable instruction, a data structure, a module of a program or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves. It should also be noted that the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, commodity or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements. It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application should be included in the scope of the claims of this application.

Claims

1. A method for sharing cloud pictures, characterized in that: Applied to cloud servers, including: Acquire picture feature information provided by a target user, where the picture feature information is used to indicate features of the picture that the target user intends to share; Perform feature matching on multiple pictures stored in the cloud by the target user based on the picture feature information to obtain at least one picture to be selected; Providing the at least one picture to be selected to the target user, and acquiring at least one target shared picture selected by the target user from the at least one picture to be selected; Building a copy prompt based on the picture feature information and / or the at least one target sharing picture, and generating a sharing copy through a large language model; A content sharing operation is performed on the at least one target sharing picture and the sharing text.

2. The method according to claim 1, characterized in that The picture feature information includes feature text; The step of performing feature matching on multiple pictures stored in the cloud by the target user based on the picture feature information to obtain at least one picture to be selected includes: Performing natural language processing on the feature text to extract the central word in the feature text; Generate a prompt text containing the central word based on a preset intention analysis template, and input it into the intention analysis large language model to obtain the intention analysis result of the target user; Screening is performed on multiple pictures stored in the cloud by the target user to obtain at least one picture to be selected that matches the intention analysis result.

3. The method according to claim 2, characterized in that The intention analysis result includes a plurality of intention feature words and sentences; The step of screening multiple pictures stored in the cloud by the target user to obtain at least one picture to be selected that matches the intention analysis result includes: Perform feature slicing on a plurality of to-be-screened pictures stored in the cloud by the target user and input them into a comparative language image pre-trained CLIP picture encoder to obtain picture encoding features of the plurality of to-be-screened pictures, wherein the picture encoding features include a three-dimensional matrix, and the three-dimensional matrix includes the number dimension of the to-be-screened pictures, the number dimension of sub-picture blocks obtained by feature slicing, and the feature dimension of the sub-picture blocks; Inputting the intention analysis result into a CLIP text encoder to obtain text encoding features of the intention analysis result, wherein the text encoding features include a two-dimensional matrix, and the two-dimensional matrix includes the quantity dimension of the intention feature words and sentences and the feature dimension of the intention feature words and sentences; A linear semantic mapping large language model is used to perform similarity matching on the text encoding features and the image encoding features to obtain at least one candidate image that matches the intent analysis result.

4. The method according to claim 3, characterized in that Performing similarity matching on the text encoding features and the picture encoding features through a linear semantic mapping large language model to obtain at least one candidate picture matching the intention analysis result, including: Mapping the image encoding feature to a semantic space matching the text encoding feature through a linear semantic mapping large language model to obtain a mapping feature; Performing a matching score on the mapping feature and the text encoding feature to obtain a matching score result, wherein the matching score result indicates the matching degree between each to-be-screened picture and the intention analysis result; At least one to-be-selected picture matching the intention analysis result is determined from a plurality of to-be-screened pictures based on the matching score result.

5. The method according to claim 4, characterized in that Performing a matching score on the mapping feature and the text encoding feature to obtain a matching score result, wherein the matching score result indicates the matching degree between each to-be-screened picture and the intention analysis result, including: Determine the similarity between the target image to be filtered and the target intention feature phrase according to the average value of the similarities between multiple sub-image blocks of the target image to be filtered among the multiple images to be filtered and the target intention feature phrases included in the multiple intention analysis results; Construct a similarity result matrix based on the similarity between each image to be screened and each intention feature phrase; Determine the picture similarity values ​​corresponding to the plurality of intention feature phrases respectively based on the similarity result matrix, wherein the picture similarity value corresponding to the intention feature phrase is the average of the similarities of the respective similar pictures in the similar picture group, and the similar picture group corresponding to the intention feature phrase includes a first preset number of similar pictures having the highest similarity to the intention feature phrase; A matching score corresponding to each image to be screened is determined based on the image similarity value and the similarity result matrix, wherein the matching score represents a comprehensive matching degree between the corresponding image to be screened and each intention feature phrase in the intention analysis result.

6. The method according to any one of claims 1 to 5, characterized in that: Constructing a copy prompt based on the picture feature information and / or the at least one target sharing picture, and generating a sharing copy through a large language model, including: Inputting the at least one target shared picture into a picture description generation model to obtain a plurality of picture description sentences; Scoring the plurality of picture description sentences based on a first preset scoring rule to determine a second preset number of target picture description sentences; Constructing a copy prompt based on the picture feature information and / or the target picture description sentence; Generate a large language model from the copy prompt input sentence copy to obtain multiple copy sentences to be selected; Scoring the plurality of candidate copywriting sentences based on a second preset scoring rule to determine a third preset number of target copywriting sentences; Generate a sharing copy based on the third preset number of target copy sentences.

7. The method according to claim 6, characterized in that The picture description generation model includes a visual feature processing module, a label attribute processing module, a matching attribute processing module and a description generation module; The at least one target shared picture is input into the picture description generation model to obtain a plurality of picture description sentences, including: Performing visual encoding on the input image through the visual feature processing module, and performing self-attention calculation on the visual encoding result to obtain visual features; Extracting label attribute data from the input image through the label attribute processing module, performing text encoding on the label attribute data to obtain image text features, and performing cross attention processing on the visual features and the image text features to obtain image attention features; Performing text encoding on at least one attribute feature matched by the input image through the matching attribute processing module to obtain an attribute text feature, and performing self-attention calculation on the attribute text feature to obtain an attribute attention feature; The description generation module performs cross-attention processing on the attribute attention features and the image attention features connected based on feature dimensions to obtain joint features, and generates multiple image description sentences through a feedforward layer.

8. The method according to claim 7, characterized in that Scoring the plurality of picture description sentences based on a first preset scoring rule to determine a second preset number of target picture description sentences includes: Performing text encoding on a first picture description sentence to obtain a description sentence text feature, wherein the first picture description sentence is any one of the multiple picture description sentences; Determining a first matching degree score between the first picture description statement and the plurality of intention feature words and sentences; Determine a second matching score between the description sentence text feature and the image text feature; Determine a sentence score of the first picture description sentence based on the first matching degree score and the second matching degree score; A second preset number of picture description sentences with high sentence scores are determined as target picture description sentences.

9. The method according to claim 6, characterized in that Constructing a copy prompt based on the image feature information and / or the target image description sentence includes: Generate a copywriting prompt based on the target image description sentence, the specified style type label and the image feature information; The step of scoring the plurality of candidate copywriting sentences based on the second preset scoring rule to determine a third preset number of target copywriting sentences includes: Performing weighted scoring on the plurality of candidate text sentences based on at least one of a sentence comfort feature, a sentence fluency feature, and a style type feature to obtain a text sentence score for each candidate text sentence; A third preset number of candidate copywriting sentences with high copywriting sentence scores are determined as target copywriting sentences.

10. The method according to claim 6, characterized in that Performing a content sharing operation on the at least one target sharing picture and the sharing copy includes: Acquire at least one associated user group of the target user, wherein the associated user group includes picture-associated users, and any picture-associated user has an associated relationship with the target user in the plurality of pictures stored in the cloud; The at least one target sharing picture and the sharing text are shared with a target associated user group selected by the target user.

Images