Received image production method, electronic equipment and system

Abstract

The invention discloses a memory image production method, electronic equipment and a memory image production system, relates to the technical field of terminals, and can solve the problem that memory influence is not related to audio and video elements and personalized contents of a user. According to the scheme, analysis and reasoning can be automatically carried out based on the content abstract of the target image element and the user portrait generation capability used for representing the personalized content of the user, the related materials are selected from the material library, and the image template used for indicating the multi-modal information is determined; and obtaining a memory image capable of matching the personalized content of the image element and the personalized content related to the user based on the image template. Moreover, according to the scheme, the links needing manual participation of the user are few, and even manual participation of the user is not needed, so that the operation of the user can be greatly simplified. Besides, when the scheme is applied to an end-cloud interaction architecture, risks such as privacy or security can be avoided, the overall processing efficiency is improved, and meanwhile, the requirement for the processing performance of the end-side electronic equipment is lowered.

Description

[0001] This application is a divisional application. The application number of the original application is 202410807630.3, and the original application date is June 20, 2024. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a memory image production method, electronic device and system. Background Art

[0003] As the pace of life becomes faster and faster, people's perception of time becomes more and more profound, so they are eager to keep the beautiful moments in life and relive the memories of the past. With the development of technology, electronic devices such as cameras and smartphones have been widely used in everyone's life applications. Users have become accustomed to capturing the wonderful moments in life through the camera capabilities of electronic devices, and recording every bit of the user's life through the album application in electronic devices.

[0004] At present, in addition to providing users with functions such as image / video browsing, image / video management (such as deletion, moving storage location, etc.), and image / video editing (such as cropping, filtering, beautification, graffiti, etc.), album applications can also provide a memory image generation function based on image and / or video integration. However, the existing memory image generation function requires users to manually operate many links, and the templates are relatively simple. Therefore, the memory image production process and the final display effect cannot bring users a particularly good user experience. Summary of the invention

[0005] The present application provides a memory image production method, electronic device and system, which can solve the problem that the generation of memory images requires a lot of manual operations by the user and the display is single, and obtain memory images with rich content that can match personalized content related to image elements and personalized content related to users, thereby improving user experience.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, a method for producing a recall image is provided, the method comprising: determining a target image element from a plurality of stored image elements; obtaining a content summary and a user portrait of the target image element, wherein the user portrait is used to characterize the user's personalized content; and determining an image template based on materials in a material library according to the content summary and the user portrait.

[0008] As an example, the image elements include pictures and / or videos. The solution provided in the first aspect above automatically performs analysis and reasoning based on the content summary of the target image element and the user portrait generation capability for characterizing the user's personalized content, selects relevant materials from the material library and determines an image template for indicating multimodal information, so as to obtain a recall image based on the image template. In this solution, since the multimodal information indicated by the image template is determined based on the content summary and the user portrait, it can match the specific situation of the image element corresponding to the content summary and one or more of the user's interests, preferences, habits, etc., so the recall image obtained based on the image template can match the personalized content of the image element and the user-related personalized content. In addition, this method requires very few manual user participation links, or even no manual user participation, so it can greatly simplify the user's operation.

[0009] As a possible implementation, the method further includes: filling content according to the image template to obtain a memory image. In this way, user operations can be simplified, and a memory image with rich content that can match personalized content related to image elements and personalized content related to users can be obtained, thereby improving user experience.

[0010] As a possible implementation, the method further includes: accepting the user's modification of the image template to obtain a modified image template; and filling the content according to the modified image template to obtain a memory image. In this way, more flexible user-defined modification can be provided, further improving the matching degree of the memory image with the user's interests, preferences or habits.

[0011] As a possible implementation, the above method is applied to an electronic device; or, the above determination of the target image element among the stored multiple image elements, and obtaining the content summary and user portrait of the target image element are performed by the electronic device; the above determination of the image template based on the material in the material library according to the content summary and user portrait is performed by the server; the above method also includes: the electronic device sends the content summary and user portrait to the server; after determining the image template, the server sends the image template to the electronic device. In this way, the flexibility and adaptability of the solution can be improved, and it is convenient to realize the recall image production process according to the actual implementation architecture and device capabilities.

[0012] As a possible implementation, the above determination of the target image element from the stored multiple image elements includes: taking at least one image element selected by the user from the stored multiple image elements as the target image element; or, determining multiple image elements with preset correlations from the stored multiple image elements as the target image elements, wherein the preset correlations include any one or more of the following: time correlation, location correlation, season correlation, object correlation, theme correlation, and description correlation. In this way, it is possible to support the triggering of memory image production in a variety of scenarios, such as active triggering by the user or automatic triggering by the device, thereby expanding the usage scenarios of the solution; in addition, in the scenario of automatic triggering by the device, it is also possible to produce memory images for image materials with correlations, thereby highlighting the theme of the memory images.

[0013] As a possible implementation, the above-mentioned obtaining of the content summary of the target image element includes: obtaining the content summary according to the original information of the target image element and / or the recognition result of the target image element. In this way, the matching degree between the obtained recall image and the personalized content related to the image element can be further improved by analyzing the target image element; in addition, the original information related to the image element in this solution will not be used as input for obtaining the image template, so even if the image template is obtained by the server, it will not bring privacy or security risks to the user.

[0014] As a possible implementation, the user portrait is obtained by: obtaining the user portrait based on the records or application data of the user using the application, wherein the user portrait is used to reflect one or more of the following aspects of the user: interests, preferences, and habits. In this way, the matching degree between the obtained memory image and the user-related personalized content is improved, and the content richness of the memory image is improved.

[0015] As a possible implementation, the material library includes one or more of the following: layout templates, text, pictures, videos, stickers, graffiti, filters, styles, music, and special effects. In this way, it is possible to obtain memory images with richer content.

[0016] As a possible implementation, the above-mentioned method of determining the image template based on the material in the material library according to the content summary and the user portrait includes: obtaining prompt information according to the content summary and the user portrait; analyzing and reasoning the prompt information, selecting relevant materials from the material library and determining the image template. In this way, it is possible to support the determination of the image template based on the content summary and the user portrait through a large model with reasoning ability, so that the image template matches the personalized content of the image element and the personalized content related to the user.

[0017] As a possible implementation, the above-mentioned image template is used to indicate multimodal information, which includes the form of the image template, the layout position of the target image element, the display size of the image element, and the theme of the memory image; wherein the form of the image template includes video, slide or picture collection. In this way, it is possible to support the acquisition of memory images in various forms, meeting the various memory image viewing needs of users.

[0018] As a possible implementation, the above-mentioned image template is also used to indicate any one or more of the following: the text and text position of the target image element, the background music name, the background music style, the background music playing time, the background music playing mode, the filter type, style, special effect type, transition effect, stickers and / or graffiti and the corresponding layout position. In this way, a memory image with rich content that can match the personalized content related to the image element and the personalized content related to the user can be obtained, thereby improving the user experience.

[0019] Exemplarily, the background music is played in a manner such as once or in a loop.

[0020] As a possible implementation, the above image template is also used to indicate the text and text position of the target image element, as well as one or more of the following text items: font, color, and font size. In this way, a memory image that can match the personalized content related to the image element and the personalized content related to the user can be obtained, thereby improving the user experience.

[0021] As a possible implementation, the above image template is not filled with relevant materials, and the above filling content according to the image template to obtain the recall image includes: editing relevant materials and layout target image elements according to the image template to obtain the recall image; or, the above image template is already filled with relevant materials, and the above filling content according to the image template to obtain the recall image includes: layout target image elements according to the image template to obtain the recall image. In this way, the flexibility and adaptability of the solution can be improved, and the recall image production process can be realized according to the actual equipment capabilities.

[0022] In a second aspect, a method for producing a recall image is provided, which can be applied to a server, and the method includes: receiving a content summary and a user portrait of a target image element from an electronic device, wherein the user portrait is used to characterize the user's personalized content; determining an image template based on materials in a material library according to the content summary and the user portrait; and sending the image template to the electronic device.

[0023] As an example, the image element includes a picture and / or a video.

[0024] The solution provided in the second aspect above, through the content summary based on the target image element and the user portrait generation capability for characterizing the user's personalized content, automatically performs analysis and reasoning, selects relevant materials from the material library and determines the image template for indicating multimodal information, so as to obtain the recall image based on the image template. In this solution, since the multimodal information indicated by the image template is determined based on the content summary and the user portrait, it can match the specific situation of the image element corresponding to the content summary and one or more of the user's interests, preferences, habits, etc., so the recall image obtained based on the image template can match the personalized content of the image element and the user's related personalized content. In addition, this method requires very few manual user participation links, or even no manual user participation, so it can greatly simplify the user's operation.

[0025] As a possible implementation, the material library includes one or more of the following: layout templates, text, pictures, videos, stickers, graffiti, filters, styles, music, and special effects. In this way, it is possible to obtain memory images with richer content.

[0026] As a possible implementation, the above-mentioned method of determining the image template based on the material in the material library according to the content summary and the user portrait includes: obtaining prompt information according to the content summary and the user portrait; analyzing and reasoning the prompt information, selecting relevant materials from the material library and determining the image template. In this way, it is possible to support the determination of the image template based on the content summary and the user portrait through a large model with reasoning ability, so that the image template matches the personalized content of the image element and the personalized content related to the user.

[0027] As a possible implementation, the above-mentioned image template is used to indicate multimodal information, which includes the form of the image template, the layout position of the target image element, the display size of the image element, and the theme of the memory image; wherein the form of the image template includes video, slide or picture collection. In this way, it is possible to support the acquisition of memory images in various forms, meeting the various memory image viewing needs of users.

[0028] As a possible implementation, the above-mentioned image template is also used to indicate any one or more of the following: the text and text position of the target image element, the background music name, the background music style, the background music playing time, the background music playing mode, the filter type, style, special effect type, transition effect, stickers and / or graffiti and the corresponding layout position. In this way, a memory image with rich content that can match the personalized content related to the image element and the personalized content related to the user can be obtained, thereby improving the user experience.

[0029] As a possible implementation, the above image template is also used to indicate the text and text position of the target image element, as well as one or more of the following text items: font, color, and font size. In this way, a memory image that can match the personalized content related to the image element and the personalized content related to the user can be obtained, thereby improving the user experience.

[0030] According to a third aspect, an electronic device is provided, comprising: a memory for storing computer program instructions; and a processor for executing the computer program instructions to support the electronic device in implementing a method as in any possible implementation of the first aspect.

[0031] In a fourth aspect, a server is provided, comprising: a memory for storing computer program instructions; and a processor for executing the computer program instructions to support the electronic device to implement the method in any possible implementation manner of the second aspect.

[0032] In a fifth aspect, a memory image production system is provided, comprising: an electronic device for implementing the method in any possible implementation of the first aspect; and a server for implementing the method in any possible implementation of the second aspect.

[0033] In a sixth aspect, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method in any possible implementation of the first aspect or the second aspect is implemented.

[0034] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to implement a method in any possible implementation of the first aspect or the second aspect.

[0035] In an eighth aspect, a chip system is provided, the chip system comprising a processing circuit and a storage medium, wherein the storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the method in any possible implementation of the first aspect or the second aspect is implemented. The chip system may be composed of a chip, or may include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is an example diagram of the production process of a common memory image;

[0037] Figure 2 This is another example of the process of making a common memory image;

[0038] Figure 3 Two system architecture schematic diagrams provided for embodiments of the present application;

[0039] Figure 4 Schematic diagram of two functional modules for producing memory images provided in an embodiment of the present application;

[0040] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0041] Fig. 6A A flow chart of a method for producing a memory image provided in an embodiment of the present application;

[0042] Figure 6B A schematic diagram of original information of a target image element provided in an embodiment of the present application;

[0043] Figure 6C A schematic diagram of a method for obtaining a user portrait provided in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of a user interface (UI) in a memory image production process provided by an embodiment of the present application;

[0045] Figure 8 A flowchart of making a memory image based on a picture provided in an embodiment of the present application;

[0046] Fig. 9 A schematic diagram of a memory image generation effect provided in an embodiment of the present application;

[0047] Fig.10 These are two example images of memory images provided in the embodiments of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0049] In the following, the terms "first", "second", etc. are only used to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. For example, if the described object is a "field", the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields", and the "first" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the described object is a "level", the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of described objects is not limited by the ordinal number, and can be one or more. Taking the "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes can be the same or different. For example, if the described object is a "device", the "first device" and the "second device" can be the same type of device or different types of devices. For another example, if the described object is "information", the "first information" and the "second information" can be information of the same content or information of different contents. In summary, the use of prefixes such as ordinal numbers to distinguish the described objects in the embodiments of the present application does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and no unnecessary limitation should be constituted due to the use of such prefixes.

[0050] Furthermore, in the embodiments of the present application, "connection" may be a direct connection or an indirect connection; in addition, it may refer to an electrical connection or a communication connection; for example, the connection between two electrical components A and B may refer to a direct connection between A and B, or may refer to an indirect connection between A and B through other electrical components or connecting media, or may refer to an indirect connection between A and B through other communication devices or communication media, as long as communication between A and B can be achieved.

[0051] As described in the background art, the existing memory image generation function requires many manual operations by the user, and the template is relatively simple. Therefore, the memory image production process and the final display effect cannot bring a particularly good user experience.

[0052] For example, please refer to Figure 1 , Figure 1 FIG. 1 shows an example of a common process of making a memory image. Figure 1 As shown, after receiving the user's operation of selecting multiple pictures and / or videos on the thumbnail interface 100 of the album application and triggering the memory image making function, the electronic device displays the pictures and / or videos selected by the user ( Figure 1 Take pictures 1, 2, ..., 9 as an example) to integrate them into videos, slides or albums and other memory images, and provide them to users Figure 1 The editing items on the interface 101, such as managing the memory image title, modifying / adding filters and background music, and managing / modifying elements (such as reselecting pictures and / or videos, modifying the order of pictures and / or videos, modifying the position of pictures and / or videos, etc.), are used by the user to manage and personalize the memory images. After completing the management and personalization of the memory images, the electronic device completes the creation of the memory images and displays them to the user. Figure 1 The interface 102 shown is used for the user to play the recall image by clicking the play button 103 .

[0053] It can be seen that based on Figure 1 In the process of making memory images by the method shown, the user needs to select pictures and / or videos, trigger the making of memory images, and manage and personalize the memory images. Therefore, there are many steps that require the user's manual participation, and the memory images finally made are highly dependent on the user's editing level and the user's knowledge and aesthetic reserves of language, music, light and shadow, etc. Figure 1 The templates used in the methods shown usually only involve the ordering, display position, and display size of pictures and / or videos, and do not take into account the personalized content related to the pictures / videos or users. Therefore, if the user's editing skills and knowledge and aesthetics of language, music, light and shadow are limited, it is often difficult to obtain a display effect that satisfies the user. In addition, Figure 1 In the method shown, similar templates are used to produce memory images for different pictures / videos or for different users, which will result in the memory images finally produced being monotonous and difficult to reflect personalized content related to the pictures / videos or users.

[0054] In order to simplify the user's operations during the production of memory images and obtain memory images that can reflect personalized content related to pictures / videos, in some embodiments, the electronic device can automatically generate memory images based on information such as the shooting time and shooting location of the pictures and / or videos, and use pictures and / or videos that meet certain conditions as production materials for the memory images.

[0055] For example, please refer to Figure 2 , Figure 2 Another common example of the process of making a memory image is shown in FIG. Figure 2 As shown, the electronic device can automatically Figure 2The photo album application in the interface 200 includes information such as the shooting time and shooting location of each picture and / or video, and selects pictures and / or videos that meet a certain time condition (such as the New Year holiday) as materials for making a memory image. According to a preset template, the selected pictures and / or videos are integrated into a memory image such as a video, a slide show or an album with the title "Happy New Year", and the memory image is provided to the user. Figure 2 The memory image preview interface on the interface 201 is shown. Figure 2 As shown in the middle interface 201, the electronic device can also Figure 2 The timeline preview window on the interface 201 shows the shooting time and location of each picture and / or video, and provides users with editing items such as clipping, audio, text editing, special effects, transition effects, filters, etc. for memory images, so that users can perform personalized editing of memory images.

[0056] It can be seen that based on Figure 2 In the process of making a memory image by the method shown in the figure, the user does not need to select a picture / video and trigger the memory image making, so compared with Figure 1 The method shown simplifies the user's operation; and when making a memory image, the electronic device can automatically determine the theme of the memory image according to the shooting time and shooting location of the picture / video, and associate the shooting time of the picture / video in the memory image, so compared with Figure 1 The method shown enriches the personalized content of the memory image. However, Figure 2 The personalized content of the memory images produced by the method shown is still relatively simple. The audio, text editing, special effects, transition effects, filters, etc. are still highly dependent on the user's editing level and the user's knowledge and aesthetic reserves of language, music, light and shadow. If the user's editing level and knowledge and aesthetic reserves of language, music, light and shadow are limited, it is often difficult to obtain a display effect that satisfies the user; and, Figure 2 The method shown strongly relies on the user's location permission being enabled when taking pictures / videos, which poses a privacy or security risk. In addition, Figure 2 In the method shown, similar templates are still used for different pictures / videos or for the memory images of different users. Figure 1 The method shown will also result in the final produced memory images being monotonous.

[0057] In some embodiments, in order to further enrich the form and content of the memory image, a memory image generation method based on a large model is adopted. Among them, the large model can provide richer image templates and more automated and intelligent memory image production through training, learning and artificial intelligence (AI) calculation processes. For example, the large model can automatically complete image template selection, theme editing, copywriting editing, background music addition, filter addition, special effect addition, transition effect addition, etc. according to multiple pictures and / or videos, and can generate rich memory images without user intervention. However, due to the large amount of materials required, the strong learning and computing power required, etc., the large model is usually deployed on the server, such as the cloud side server. Therefore, when making a memory image, the electronic device needs to send the picture / video and its related original information (such as shooting time, shooting location, etc.) to the server, which poses privacy or security risks. In addition, although this solution can enrich the form and content of the memory image, its form and content mainly rely on the learning and computing power of the large model, and does not take into account the personalized content related to the user, so the effect of the memory image finally produced still has a lot of room for improvement.

[0058] In order to solve the above problems existing in the conventional memory image production scheme, the embodiment of the present application provides a memory image production method, which can automatically obtain the image template and use the image template to produce a memory image that can match the personalized content related to the image element (picture and / or video, etc.) and the personalized content related to the user based on the content summary generation capability and user portrait generation capability of the electronic device, and the image template production capability of the large model based on the content summary, user portrait, etc. and rich materials. The method requires very few manual user participation links, and even does not require manual user participation, so it can greatly simplify the user's operation; in addition, the memory image finally produced by this method can not only generate a memory image with rich content, but also the theme, copywriting, background music, filter, style, special effects, transition effects and other contents can match the personalized content related to the image element and the personalized content related to the user, so that the memory image with richer content, more natural effect and more in line with the actual scene can be obtained; in addition, the original information related to the image element in this scheme will not be used as the input of the large model, so even if the large model is deployed on the server, it will not bring privacy or security risks.

[0059] Among them, in the embodiment of the present application, the content summary can represent but is not limited to any one or more of the following information of the image elements (such as pictures and / or videos): objects (such as people, things, etc.), styles, filters, descriptions (such as introductions, tags, categories, etc. added by users), locations, seasons, times, etc.

[0060] User portraits can represent but are not limited to one or more personalized content such as user's interests, preferences, habits, etc. For example, user portraits can represent but are not limited to any one or more of the following information of the user: favorite music, favorite music style, favorite singer, favorite singer / song era, hobbies (such as traveling, sports, reading, etc.), personality (such as sunny and cheerful, quiet, etc.), habits, special time points (such as birthdays, anniversaries of certain events, etc.), etc.

[0061] In some embodiments, the memory image production method provided in the embodiments of the present application can be applied to the architecture of end-cloud interaction. For example, the method can be applied to Figure 3 In the architecture shown in (a) of FIG. 1 , an electronic device and a server (such as a cloud server) are provided, wherein the electronic device is responsible for storing image elements (such as pictures and / or videos), generating a content summary based on at least one stored image element, generating a user portrait based on the user's history of using the electronic device, and providing the content summary and the user portrait to the server, so that the server can obtain an image template based on the content summary, the user portrait and the rich materials through a large model. And, Figure 3 The electronic device shown in (a) is also used to fill the image template determined by the server with relevant image elements, and automatically produce a memory image that can match the personalized content related to the image elements and the personalized content related to the user.

[0062] In some embodiments, the memory image production method provided in the embodiments of the present application can be applied to a single device architecture. For example, the method can be applied to Figure 3 In the electronic device shown in (b), the electronic device can first generate a content summary based on at least one stored image element, and generate a user portrait based on the user's historical record of using the electronic device, and then obtain an image template based on the content summary, user portrait and rich materials through a large model, and finally fill the image template with relevant image elements to automatically produce a recall image that can match the personalized content related to the image elements and the personalized content related to the user.

[0063] There is no specific limitation on the specific architecture of the memory image production method provided in the embodiment of the present application, and it may depend on the specific device type, device function, device computing power, application scenario, etc.

[0064] As an example, see Figure 4 , Figure 4 Respectively Figure 3 (a) and Figure 3 Taking the architecture shown in (b) as an example, two functional module schematic diagrams for producing recall images provided in an embodiment of the present application are shown.

[0065] in, Figure 4(a) in FIG. 1 shows a schematic diagram of a functional module for producing a memory image in a terminal-cloud collaborative architecture. Figure 4 As shown in (a), the electronic device 310 may include a storage module 310-1, an element analysis module 310-2, a user portrait extraction module 310-3, a memory module 310-4, a recall image processing module 310-5 and a communication module 310-6.

[0066] in, Figure 4 The storage module 310 - 1 shown in (a) is used to store image elements (such as pictures and / or videos) and store the user's historical records of using the electronic device.

[0067] Figure 4 The element analysis module 310-1 shown in (a) is used to generate a content summary based on at least one image element (such as a picture and / or video) stored in the storage module 310-1. For example, the element analysis module 310-1 may include a neural-network processing unit (NPU) and / or a graphics processing unit (GPU) with image recognition capabilities, executable AI model reasoning capabilities and content summary generation capabilities, and is used to generate a content summary about the at least one image element (such as a picture and / or video) stored in the storage module 310-1 through image recognition, analysis and reasoning processes.

[0068] Exemplarily, the content summary may represent but is not limited to any one or more of the following information of an image element (such as a picture and / or video): object (such as a person, thing, etc.), style, filter, description (such as a user-added introduction, tag, classification, etc.), location, season, time, and the like.

[0069] Figure 4The user portrait extraction module 310-3 shown in (a) is used to obtain user usage records, generate user portraits, etc. Obtaining user usage records, such as obtaining records of user operations on electronic device applications and functions during the user's use of the electronic device, such as obtaining song information played by the user using the music application, obtaining memo information set by the user, obtaining calendar events set by the user, etc.; after obtaining the user usage records, the user portrait extraction module 310-3 can store the user usage records in the memory module 310-4. Generating user portraits, such as generating user portraits that can reflect one or more of the user's interests, preferences, habits, etc., based on the user usage records stored in the memory module 310-4 through sorting, synthesis, and refinement, for example, the user portrait extraction module 310-3 can generate user portraits based on the user usage records stored in the memory module 310-4 on a regular basis (such as every Monday, at the end of each month, etc.); after generating the user portraits, the user portrait extraction module 310-3 can store the user portraits in the memory module 310-4.

[0070] Exemplarily, a user portrait may represent, but is not limited to, one or more personalized contents such as the user's interests, preferences, habits, etc. For example, a user portrait may represent, but is not limited to, any one or more of the following information of the user: favorite music, favorite music style, favorite singer, favorite singer / song era, hobbies (such as traveling, sports, reading, etc.), personality (such as sunny and cheerful, quiet, etc.), habits, special time points (such as birthdays, anniversaries of certain events, etc.), etc.

[0071] Figure 4 The memory module 310-4 shown in (a) is used to store the user usage records and user portraits obtained by the user portrait extraction module 310-3. As an example, the memory module 310-4 may include a physical storage medium and a logical storage engine to support the memory module 310-4 to save the user usage records and user portraits.

[0072] Figure 4 The memory image processing module 310-5 shown in (a) is used to fill in relevant image elements according to the image template obtained by the server 320, and obtain memory images such as videos, slides or albums that can match the personalized content related to the relevant image elements and the user's personalized content.

[0073] Figure 4The communication module 310-6 shown in (a) is used to send the content summary and the user portrait to the server 320, and receive the image template from the server 320. For example, the communication module 310-6 can periodically send the content summary of the image elements (such as pictures and / or videos) in the first time period (such as the past week, the past month, etc.) and the user portrait in the second time period (such as the last week, the last month, etc.) to the server 320; or, the communication module 310-6 can send the content summary of some image elements (such as pictures and / or videos selected by the user, etc.) and the user portrait in the second time period (such as the last week, the last month, etc.) to the server 320 based on certain trigger conditions.

[0074] like Figure 4 As shown in (a) in FIG. 1 , the server 320 may include a communication module 320 - 1 , an image template acquisition module 320 - 2 and a material library 320 - 3 .

[0075] in, Figure 4 The communication module 320 - 1 shown in (a) receives the content summary and user portrait from the electronic device 310 , and sends the image template acquired by the image template acquisition module 320 - 2 to the electronic device 310 .

[0076] Figure 4 The material library 320-3 shown in (a) is used to manage and store materials used to generate image templates, such as one or more multi-modal materials including but not limited to layout templates, text, pictures, videos, stickers, graffiti, filters, styles, music, special effects, etc., to enhance the content richness of image templates. The layout template is used to specify but not limited to the layout position, display size, etc. of image elements such as pictures and / or videos.

[0077] In some embodiments, Figure 4 The material library 320-3 shown in (a) can also automatically update materials, such as regularly synchronizing with the cloud material platform to update materials, updating materials when new materials are input, triggering material updates based on preset trigger conditions, etc., without limitation.

[0078] Figure 4 The image template acquisition module 320-2 shown in (a) is used to obtain an image template including multimodal information based on the content summary and user portrait from the electronic device 310, using the materials in the material library 320-3 that can match the content summary and the user portrait, and provide the image template to the electronic device 310 for subsequent generation of recall images such as videos, slides or photo albums.

[0079] Among them, the image template obtained by the image template acquisition module 320-2 is used to indicate the form of the image template (such as video, slide or picture collection, etc.), the layout position of the image element corresponding to the content summary, the display size of the image element corresponding to the content summary, the recollection image theme, and any one or more of the following: the text and text position of the image element corresponding to the content summary, the background music name, background music style, background music playback time, background music playback method (such as one-time playback or loop playback, etc.), filter type, style, special effect type, transition effect, stickers and / or graffiti and corresponding layout position, etc. In some embodiments, if the image template is used to indicate the text of the image element corresponding to the content summary, the image template may also be used to indicate one or more of the font, color, font size, etc. of the text. Among them, the content indicated by the above-mentioned image template is determined based on the content summary and the user portrait, so it can match the specific situation of the image element corresponding to the content summary and one or more of the user's interests, preferences, habits, etc.

[0080] As an example, Figure 4 The image template acquisition module 320-2 shown in (a) can perform a first processing on the content summary and user portrait from the electronic device 310, and then analyze and infer, and further determine the relevant materials and image templates. For example, the first processing includes splicing the content summary and user portrait based on a preset rule. For another example, the first processing includes optimizing the content summary and / or user portrait and then splicing the content based on a preset rule.

[0081] As an example, Figure 4 The image template acquisition module 320-2 shown in (a) may include a GPU or NPU with AI model reasoning capabilities. The image template acquisition module 320-2 can use the obtained results after the first processing as input to the GPU or NPU with AI model reasoning capabilities for analysis and reasoning, as well as further related material determination and image templates.

[0082] Figure 4 (b) in FIG. 1 shows a schematic diagram of a functional module for producing a memory image in a single device architecture. Figure 4 As shown in (b), the electronic device 310 may include a storage module 310-1, an element analysis module 310-2, a user portrait extraction module 310-3, a memory module 310-4, a recall image processing module 310-5, an image template acquisition module 310-7 and a material library 310-8.

[0083] Among them, about Figure 4The functions of the storage module 310-1, the element analysis module 310-2, the user portrait extraction module 310-3, the memory module 310-4 and the memory image processing module 310-5 shown in (b) can be referred to in the above description. Figure 4 (a) shows an introduction to the storage module 310-1, element analysis module 310-2, user portrait extraction module 310-3, memory module 310-4 and recall image processing module 310-5.

[0084] Figure 4 The material library 310-8 shown in (b) is used to manage and store materials used to generate image templates, such as but not limited to typesetting templates, text, pictures, videos, stickers, graffiti, filters, styles, music, special effects and other multimodal materials, so as to enhance the content richness of the image templates.

[0085] In some embodiments, Figure 4 The material library 310 - 8 shown in (b) can also automatically update materials, such as regularly synchronizing with the cloud material platform to update materials, updating materials when new materials are input, triggering material updates based on preset trigger conditions, etc., without limitation.

[0086] Figure 4 The image template acquisition module 310-7 shown in (b) is used to obtain an image template including multimodal information based on the content summary and user portrait stored in the memory module 310-4, using the materials in the material library 310-8 that can match the content summary and user portrait, and provide the image template to the recall image processing module 310-5 for subsequent generation of recall images such as videos, slides or photo albums.

[0087] Among them, the image template obtained by the image template acquisition module 310-7 is used to indicate the form of the image template (such as video, slide or picture collection, etc.), the layout position of the image element corresponding to the content summary, the display size of the image element corresponding to the content summary, the recollection image theme, and any one or more of the following: the text and text position of the image element corresponding to the content summary, the background music name, background music style, background music playback time, background music playback method (such as one-time playback or loop playback, etc.), filter type, style, special effect type, transition effect, stickers and / or graffiti and corresponding layout position, etc. In some embodiments, if the image template is used to indicate the text of the image element corresponding to the content summary, the image template may also be used to indicate one or more of the font, color, font size, etc. of the text. Among them, the content indicated by the above-mentioned image template is determined based on the content summary and the user portrait, so it can match the specific situation of the image element corresponding to the content summary and one or more of the user's interests, preferences, habits, etc.

[0088] Need to explain, Figure 4 (a) and Figure 4 (b) is only used as two examples of functional modules for making memory images. The embodiments of the present application do not limit the specific architecture and the specific functional module settings in the electronic device or server. For example Figure 4 The material library 310-8 shown in (b) may also be set in the server. Figure 4 (a) or Figure 4 The multiple modules in (b) may be integrated together or the function of a module may be completed by multiple sub-modules, depending on the specific situation.

[0089] Among them, the electronic device described in the embodiment of the present application may include but is not limited to any electronic device with image display, such as the electronic device may include but is not limited to smart phones, netbooks, tablet computers, smart drawing boards, handwriting boards, smart watches, smart bracelets, phone watches, smart glasses, smart cameras, PDAs, car computers, personal computers (personal computers, PCs), personal digital assistants (personal digital assistants, PDAs), portable multimedia players (portable multimedia players, PMPs), augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) devices, smart TVs, projection devices, or somatosensory game consoles in human-computer interaction scenarios, etc. Alternatively, the electronic device may also be other types or structures of electronic devices with image display, which are not limited in the present application.

[0090] As an example, see Figure 5 , Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0091] like Figure 5 As shown, the electronic device may include a processor 510, a memory (including an external memory interface 520 and an internal memory 521), a universal serial bus (USB) interface 530, a charging management module 540, a power management module 541, a battery 542, an antenna 1, an antenna 2, a mobile communication module 550, a wireless communication module 560, an audio module 570, a speaker 570A, a receiver 570B, a microphone 570C, an earphone interface 570D, a sensor module 580, a button 590, a motor 591, an indicator light 592, a camera 593, a display screen 594, etc.

[0092] Among them, the sensor module 580 may include but is not limited to one or more of the following: touch sensor, pressure sensor, temperature sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, ambient light sensor, bone conduction sensor, etc.

[0093] The processor 510 may include one or more processing units. For example, the processor 510 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a flight controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0094] The processor 510 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 510 is a cache memory. The memory may store instructions or data that the processor 510 has just used or cyclically used. If the processor 510 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 510, and thus improves the efficiency of the system.

[0095] In some embodiments of the present application, the processor 510 can send the application interface, memory image, etc. to the display screen 594 for display through an interface, such as a DSI interface.

[0096] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 550, wireless communication module 560, modem processor and baseband processor.

[0097] In some embodiments of the present application, the electronic device can communicate with other devices through antenna 1, antenna 2, mobile communication module 550, wireless communication module 560, modem processor and baseband processor, such as sending content summary and user portrait to the server, receiving image templates or materials from the server, etc.

[0098] The electronic device implements the display function through a GPU, a display screen 594, and an AP. The GPU is a microprocessor for image processing, connected to the display screen 594 and the AP. The GPU is used to perform mathematical and geometric calculations for drawing, rendering, or compositing graphics. The processor 510 may include one or more GPUs that execute program instructions to generate or change display information.

[0099] In some embodiments of the present application, the electronic device can implement related processing of the application interface and / or the memory image through AP, GPU, etc., and display the application interface and / or the memory image through the display screen 594.

[0100] The electronic device can realize the shooting function of image elements (such as pictures and / or videos, etc.) through ISP, camera 593, video codec, GPU, display screen 594 and AP.

[0101] The external memory interface 520 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 510 through the external memory interface 520 to implement a data storage function.

[0102] The internal memory 521 can be used to store computer executable program codes. Exemplarily, the computer program may include an operating system program and an application program. Among them, the executable program code includes instructions. The processor 510 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 521. The internal memory 521 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application program required for at least one function, etc. The data storage area may store data created during the use of the electronic device, etc. In addition, the internal memory 521 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 510 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 521, and / or the instructions stored in the memory provided in the processor.

[0103] In some embodiments of the present application, the internal memory 521 can be used to store image elements such as pictures and videos, original information related to pictures (such as shooting time, shooting location, etc.), original information related to videos (such as shooting time, shooting location, etc.), historical records of users using electronic devices, as well as content summaries and user portraits.

[0104] In some embodiments of the present application, the internal memory 521 can also be used to store materials (such as layout templates, text, pictures, videos, stickers, graffiti, filters, styles, music, special effects, etc.), image templates, etc.

[0105] The electronic device can implement audio functions such as music playing and recording through the audio module 570, the speaker 570A, the receiver 570B, the microphone 570C and the AP.

[0106] about Figure 5 The charging management module 540, power management module 541, battery 542, motor 591, indicator light 592, etc. shown in the figure can be referred to conventional technology and will not be elaborated here.

[0107] It is understandable that this application Figure 5 The illustrated structure does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0108] As an example, see Fig. 6A , Fig. 6A FIG. 1 shows a flow chart of a method for producing a memory image provided by an embodiment of the present application. Fig. 6A As shown, the method may include S601-S604:

[0109] S601: Determine a target image element among a plurality of stored image elements.

[0110] The stored image elements may include but are not limited to pictures and / or videos, etc. The stored image elements may be taken by the electronic device, or may be received by the electronic device from other devices or downloaded from the Internet, which is not limited in this application.

[0111] As a possible implementation manner, the target image element may be determined based on a user's selection operation on at least one of the plurality of stored image elements, that is, the target image element is at least one image element selected by the user from the plurality of stored image elements.

[0112] As another possible implementation manner, at least one target image element may be automatically determined from a plurality of stored image elements based on an AI model.

[0113] As an example, multiple image elements with preset correlations among multiple stored image elements may be periodically and automatically selected as target image elements.

[0114] For example, the preset correlation may include any one or more of the following: time correlation, location correlation, season correlation, object correlation, theme correlation, description correlation, etc. Exemplarily, there is time correlation, such as the shooting time / download time is within a preset time range (such as one day, one week, one month, half a year, one year, etc.), there is location correlation, such as the shooting location is within a preset range (such as a county, city, province, etc.), there is season correlation, such as the background environment belongs to the same season, there is object correlation, such as including the same or related people, things, etc., there is theme correlation, such as the theme is the same or related, and there is description correlation, such as the introduction, label, classification, etc. are the same or related.

[0115] When determining a target image element from multiple stored image elements, as a possible implementation method, an image element with a preset correlation can be identified from the multiple image elements as the target image element based on the original information of the multiple image elements and / or the recognition results of the image elements.

[0116] Taking the target image element, which is a picture or video, as an example, we can identify image elements with any one or more correlations such as time correlation, location correlation, season correlation, object correlation, theme correlation, description correlation, etc. from multiple image elements based on the original information of the picture / video, such as the shooting time, shooting location, introduction, label, classification, etc., as well as the recognition results of the object, background environment, etc. obtained after image recognition of the picture / video, and use them as target image elements.

[0117] S602: Obtain a content summary of a target image element, and obtain a user portrait for representing user personalized content.

[0118] The content summary may represent but is not limited to any one or more of the following information of the target image element: object (such as a person, thing, etc.), style, filter, description (such as a user-added introduction, tag, classification, etc.), location, season, time, etc.

[0119] As a possible implementation method, based on modules such as NPU and / or GPU that have image recognition capabilities, executable AI model reasoning capabilities, and content summary generation capabilities, a content summary of the target image element can be ultimately generated through image recognition, analysis, and reasoning processes based on the original information of the target image element and / or the recognition results of the target image element.

[0120] For example, the target image element is a picture and / or video, and the original information of the target image element includes the shooting time, shooting location, introduction, label, category, etc. of the picture and / or video. For example, please refer to Figure 6B , Figure 6BFIG. 1 shows a schematic diagram of original information of a target image element provided by an embodiment of the present application. Figure 6B As shown, the target image element is Figure 6B The original information of Picture 1, Picture 2 and Picture 3 in the album "B Scenic Spot" on the interface 600 shown includes: shooting time 10:32 am on March X1st, shooting location B Scenic Spot in City D of Province A, label "holiday, travel" and introduction "take children to visit B Scenic Spot"; the original information of Picture 2 includes: shooting time 10:56 am on March X1st, shooting location B Scenic Spot in City D of Province A, label "holiday, travel" and introduction "take children to visit B Scenic Spot"; the original information of Picture 3 includes: shooting time 11:20 am on March X1st, shooting location B Scenic Spot in City D of Province A, label "holiday, travel" and introduction "take children to visit B Scenic Spot".

[0121] As a possible implementation, the location, time, description (such as the introduction, label, classification, etc. added by the user) represented by the content summary may be determined based on the original information of the target image element and / or the recognition result of the target image element; the object (such as people, things, etc.), style, filter, season, etc. represented by the content summary may be determined based on the recognition result of the target image element. Of course, the above several possible implementations of determining the content summary are only examples. In actual applications, the basis and determination process of determining each specific content summary are not limited.

[0122] Among them, the user portrait can represent but is not limited to one or more personalized contents such as the user's interests, preferences, habits, etc. For example, the user portrait can represent but is not limited to any one or more of the following information of the user: favorite music, favorite music style, favorite singer, favorite singer / song era, hobbies (such as traveling, sports, reading, etc.), personality (such as sunny and cheerful, quiet, etc.), habits, special time points (such as birthdays, anniversaries of certain events, etc.), etc.

[0123] As a possible implementation method, a user profile that can reflect the user's personalized content, such as one or more interests, preferences, habits, etc., can be generated based on the user's usage records through sorting, integration, and refinement.

[0124] As an example, user usage records may include but are not limited to records or application data during the user's use of the application. For example, please refer to Figure 6C , Figure 6C A schematic diagram of a method for obtaining a user portrait provided by an embodiment of the present application is shown. Figure 6C As shown, user usage records are as follows: Figure 6C The call history, contact information, etc. in the call application shown in (a) above. Figure 6C The number of steps and / or exercise time in the exercise application shown in (b) in FIG. Figure 6C The audio playlist or favorite songs in the audio and video application shown in (c) above, Figure 6C (d) shows the user activity trajectory in the map application. Figure 6C (e) in the calendar application or memo application, marking special time points (such as birthdays, anniversaries of certain events, etc.), to-do items, etc., Figure 6C (f) shows the usage data of each application (such as usage frequency).

[0125] As a possible implementation, the user's favorite music, favorite music style, favorite singer, favorite singer / song era, etc. represented by the user portrait may be determined based on the audio and / or video playlists, favorite songs, etc. in the process of the user using the audio application; the special time points represented by the user portrait (such as birthdays, anniversary of certain events, etc.) may be determined based on the special time points and / or to-do items recorded by the user in the process of using the calendar application and / or memo application; the hobbies represented by the user portrait (such as travel, sports, reading, etc.) may be determined based on the frequency of the user's use of various applications; the personality represented by the user portrait (such as sunny and cheerful, quiet, etc.) may be determined based on the text posted by the user in the process of using social applications. Of course, the above several possible implementations of determining user portraits are only examples. In actual applications, the basis and determination process for determining each specific user portrait are not limited.

[0126] S603: Determine an image template based on the materials in the material library according to the obtained content summary and user portrait.

[0127] The material library is used to manage and store materials used to generate image templates. For example, the material library may include but is not limited to one or more multimodal materials such as various layout templates, text, pictures, videos, stickers, graffiti, filters, styles, music, special effects, etc. The layout template is used to specify but not limited to the layout position, display size, etc. of the target image element.

[0128] As a possible implementation method, an image template can be obtained based on the obtained content summary and user portrait using the material in the material library that can match the content summary and the user portrait. Among them, the image template is used to indicate multimodal information, such as the image template indicating the form of the image template (such as video, slide or picture collection, etc.), the layout position of the image element corresponding to the content summary, the display size of the image element corresponding to the content summary, the memory image theme, and any one or more of the following: the text and text position of the image element corresponding to the content summary, the background music name, the background music style, the background music playback time, the background music playback method (such as one-time playback or loop playback, etc.), the filter type, style, special effect type, transition effect, stickers and / or graffiti and the corresponding layout position, etc.

[0129] In some embodiments, if the image template is used to indicate the text of the image element corresponding to the content summary, the image template may also be used to indicate one or more items such as the font, color, and font size of the text.

[0130] As a possible implementation, when determining the image template, the content summary and the user portrait may be first processed, and then analysis and reasoning may be performed based on the prompt information obtained after the first processing, and relevant materials may be selected from the material library to determine the image template. For example, the first processing may include splicing the content summary and the user portrait based on preset rules. For another example, the first processing may include splicing the content based on preset rules after optimizing the content summary and / or the user portrait.

[0131] As an example, as a possible implementation method, when determining the image template, the content summary and user portrait can be first processed, and then the prompt information obtained after the first processing can be used as the input of the GPU or NPU with AI model reasoning capabilities. The GPU or NPU can perform analysis and reasoning based on the prompt information, and further determine the relevant materials and image templates.

[0132] It can be understood that since the multimodal information indicated by the image template is determined based on the content summary and the user portrait, it can match the specific situation of the target image element corresponding to the content summary and one or more of the user's interests, preferences, habits, etc.

[0133] S604: Filling the content according to the determined image template to obtain a recall image.

[0134] The form of the memory image may include but is not limited to video, slide show or photo album, etc. The specific form of the memory image is not limited in the embodiment of the present application.

[0135] As a possible implementation method, the relevant materials can be edited and the relevant target image elements can be laid out according to the determined image template to obtain a memory image such as a video, a slide show, or an album that can match the personalized content of the relevant target image elements and the user's relevant personalized content and includes multimodal information. For example, if the image template is not filled with relevant materials, the relevant materials can be edited into the image template, and the relevant target image elements can be laid out into the image template to obtain a memory image.

[0136] As a possible implementation method, the relevant target image elements can be directly laid out according to the determined image template to obtain a memory image such as a video, slide show, or album that can match the personalized content of the relevant target image elements and the user's relevant personalized content and includes multimodal information. For example, if the image template has been edited with relevant materials, the relevant target image elements can be directly laid out into the image template to obtain a memory image.

[0137] For example, please refer to Figure 7 , Figure 7 FIG. 1 is a schematic diagram of a UI interface in a memory image production process provided by an embodiment of the present application. Figure 7 (a) and Figure 7 As shown in (b) of FIG. 7 , after obtaining the image template, the interface 700 may display relevant information of the image template, such as but not limited to the form of the image template (such as video, slide or picture collection, etc.), the layout position of the target image element, the display size of the target image element, the memory image theme, the text and text position of the target image element, the name of the background music, etc. Figure 7 As shown in (a) in the figure, the user can trigger the creation of a memory image by clicking a "create memory image" button to obtain a memory image.

[0138] Need to explain, Figure 7 (a) and Figure 7 (b) in the figure only shows information of several modes in the image template. In practical applications, the categories of multimodal information displayed on the interface are not limited. Figure 7 (a) and Figure 7 The interface 700 shown in (b) may also display the background music playback duration, background music playback mode, filter type, style, special effect type, transition effect, stickers and / or graffiti and corresponding layout position, etc. of the image template.

[0139] In some embodiments, after obtaining the image template, the user can also be provided with a modification function for the multimodal information in the image template, so that the user can modify one or more of the following: the form of the image template, the layout position of the target image element, the display size of the target image element, the recollection image theme, the text and text position of the target image element, the background music name, the background music style, the background music playing time, the background music playing mode, the filter type, style, special effect type, transition effect, stickers and / or graffiti, and the corresponding layout position, and based on the image template modified by the user, the target image element layout and material editing are performed to obtain the recollection image. For example, Figure 7 As shown in (b) in the figure, users can click Figure 7 The "Modify Image Template" button on the interface 700 shown in (b) triggers the display of the image template modification interface, and makes relevant modifications on the image template modification interface, wherein Figure 7 (b) in the figure takes the example of a user modifying the theme of the recollection image to "Happy travel moments at attraction B" on interface 704 (of course, the user can also modify other contents on interface 700), and the recollection image based on the image template modified by the user is displayed to the user through interface 704.

[0140] In some embodiments, after the recall image is obtained, the recall image can be saved.

[0141] In some embodiments, after obtaining the recall image, a preview interface of the recall image may also be displayed (eg, Figure 7 The interface 701 shown in (a) and Figure 7 The interface 705 shown in (b) in FIG. 1 is used for the user to play the memory image by clicking the play button on the preview interface; Figure 7 In (a), after receiving the user's click on the play button on the preview interface 701, the playback of the recall image will be triggered through the interface 702; Figure 7 As shown in (b) in FIG. 7 , upon receiving the user's click on the play button on the preview interface 705 , the playback of the recall image will be triggered through the interface 706 .

[0142] Understandably, based on Fig. 6AThe method for producing a memory image shown in the figure automatically analyzes and infers based on the content summary of the target image element and the user portrait generation capability for representing the user's personalized content, selects relevant materials from the material library and determines the image template for indicating multimodal information, and obtains the memory image based on the image template. Since the multimodal information indicated by the image template is determined based on the content summary and the user portrait, it can match the specific situation of the image element corresponding to the content summary and one or more of the user's interests, preferences, habits, etc., so the memory image obtained based on the image template can match the personalized content of the image element and the user-related personalized content.

[0143] This method requires very little manual participation from the user, or even no manual participation from the user, thus greatly simplifying the user's operation.

[0144] Moreover, the memory images finally produced by this method are better than Figure 1 The solution shown does not take into account the personalized content related to the image / video or user, or Figure 2 The solution with a single personalized content shown can not only generate memory images with rich content, but also the themes, texts, background music, filters, styles, special effects, transition effects and other contents can match the personalized content related to the image elements and the personalized content related to the user. Therefore, it is possible to obtain memory images with richer content, more natural effects and more in line with the actual scenes, which contain user-related story context, plot and timeline, meet the personalized requirements of users, and achieve the effect of generating memory images with thousands of faces for thousands of people.

[0145] Furthermore, the method can provide a variety of different forms of memory image generation, such as video, slide show or album, etc., so it can meet the diverse needs of users and display a variety of memory images.

[0146] In addition, when this method is applied to the architecture of end-cloud interaction, the original information of the image elements in the electronic device and the records or application data of the user's use of the application will not be used as input for determining the image template. Instead, a content summary of the image elements is generated through analysis and reasoning based on the original information of the image elements and / or the recognition results of the image elements. A user portrait that can reflect the user's personalized content is generated through organization, synthesis, and refinement based on the user's usage records. The content summary and user portrait are used as input for determining the image template, thereby avoiding privacy or security risks.

[0147] In addition, when this method is applied to the architecture of end-cloud interaction, it can not only utilize the portability and real-time performance of the end-side electronic devices, but also utilize the powerful computing power and rich material resources of the large model in the cloud-side server. It can not only simplify user operations and obtain recall images with richer content, more natural effects, and more realistic scenes that contain user-related story context, plots, and timelines, but also improve the overall processing efficiency while reducing the requirements for the processing performance of the end-side electronic devices.

[0148] It should be noted that the embodiments of the present application do not limit Fig. 6A The system architecture to which the method shown is applied, e.g. Fig. 6A The method shown can be applied to Figure 3 (a) Figure 3 (b) Figure 4 (a) or Figure 4 The architecture shown in (b) is shown in Fig. 6A The method shown is applied to Figure 3 (a) or Figure 4 When the architecture shown in (a) is used, S601-S602 and S604 may be executed by the electronic device 310, and S603 may be executed by the server 320; Fig. 6A The method shown is applied to Figure 3 (b) or Figure 4 In the architecture shown in (b), S601-S604 are all executed by the electronic device 310.

[0149] As an example, when Fig. 6A The method shown is applied to Figure 4 In the architecture shown in (a), S601 may be executed by the element analysis module 310-2 of the electronic device 310, S602 may be executed by the element analysis module 310-1 and the user portrait extraction module 310-3 of the electronic device 310, S603 may be executed by the image template acquisition module 320-2 of the server 320, and S604 may be executed by the recall image processing module 310-5 of the electronic device 310.

[0150] As an example, when Fig. 6A The method shown is applied to Figure 4 In the architecture shown in (b), S601 may be executed by the element analysis module 310-2 of the electronic device 310, S602 may be executed by the element analysis module 310-1 and the user portrait extraction module 310-3 of the electronic device 310, S603 may be executed by the image template acquisition module 310-7 of the electronic device 310, and S604 may be executed by the recall image processing module 310-5 of the electronic device 310.

[0151] The following will take the image element being a picture taken by a user as an example, and combine with specific examples to introduce a flow chart of a method for producing a memory image provided in an embodiment of the present application.

[0152] For example, please refer to Figure 8 , Figure 8 Taking the architecture applied to end-cloud interaction as an example, a schematic diagram of a process of making a memory image based on a picture provided by an embodiment of the present application is shown. Fig. 6A The S601 shown may include Figure 8 As shown in S801, the above Fig. 6A S602 shown may include Figure 8 As shown in S802-S804, the above Fig. 6A S603 shown may include Figure 8 As shown in S805-S807, the above Fig. 6A The illustrated S604 may include Figure 8 S808-S809 are shown. For the related introduction of S801-S809, please refer to the above description of S601-S604.

[0153] like Figure 8 As shown, the production of memory images can be achieved based on S801-S809:

[0154] S801: The electronic device saves multiple pictures taken by the user.

[0155] Among them, the multiple pictures taken by the user are the target image elements mentioned above.

[0156] For example, during a vacation when the user takes his children to travel to Province A, the user takes some pictures of people and scenery. After the user completes taking the pictures, the electronic device saves the multiple pictures taken by the user, wherein the multiple pictures saved by the electronic device include original information of the pictures, such as shooting time, shooting location, etc.

[0157] In some embodiments, if the user adds a description or label to the picture, such as "Take the children to visit attraction B", and / or the user archives the picture to an album named "Attraction B", the electronic device may also save the corresponding description, label or category as part of the original information.

[0158] Exemplarily, the multiple images include Figure 6BIn the album "B Scenic Spot" on the middle interface 600, pictures 1, 2, 3, etc. are shown, among which picture 1 is a picture taken on March X1, at the water town of B Scenic Spot, D City, A Province, showing a scene of a small bridge and flowing water; picture 2 is a picture taken on March X1, at B Scenic Spot, D City, A Province, showing a child feeding fish by a pond surrounded by rockery; picture 3 is a picture taken on March X1, at B Scenic Spot, D City, A Province, showing a child in a pavilion on a rockery. Among them, pictures 1, 2, and 3 are labeled "holiday, travel" and the introduction is "take children to visit B Scenic Spot".

[0159] S802: The electronic device performs image analysis on the multiple images to obtain recognition results.

[0160] As an example, an electronic device can perform object recognition, comprehensive analysis and reasoning on multiple pictures to obtain recognition results based on modules such as NPU and / or GPU that have image recognition capabilities and executable AI model reasoning capabilities.

[0161] For example, the multiple pictures include Picture 1, Picture 2 and Picture 3, where Picture 1 is a picture of a scene of a small bridge and flowing water in a water town taken by a user, Picture 2 is a picture of a child in a rockery pavilion taken by the user, and Picture 3 is a picture of a child feeding fish by a pond surrounded by rockery taken by the user. After object recognition, comprehensive analysis and reasoning, the electronic device can recognize the small bridge and flowing water and other landscapes in Picture 1, the rockery, pavilion and other landscapes and the child in Picture 2, and the rockery, pond and the scene of the child feeding fish in Picture 3.

[0162] S803: The electronic device obtains content summaries of the multiple pictures according to the original information of the multiple pictures and the recognition results.

[0163] As a possible implementation manner, the electronic device may obtain a content summary of each picture based on original information such as shooting time, shooting location, introduction, label, classification, etc. of the multiple pictures and the recognition results.

[0164] For example, the multiple pictures include picture 1, picture 2, and picture 3, where picture 1 is a picture of a small bridge and flowing water in a water village taken by a user, picture 2 is a picture of a child in a pavilion among rockery taken by the user, and picture 3 is a picture of a child feeding fish beside a pond surrounded by rockery taken by the user. After comprehensive analysis and reasoning, the electronic device can obtain the following content summaries of picture 1, picture 2, and picture 3 respectively:

[0165] Picture 1: Jiangnan Water Village, ancient and natural, winding paths with small bridges and flowing water, no people;

[0166] Picture 2: Jiangnan Water Village, ancient and natural, well-arranged rockery pavilions, and a little boy standing;

[0167] Picture 3: Jiangnan water town, ancient and natural style, a pond surrounded by rockery, and a little boy feeding fish.

[0168] S804: The electronic device obtains a user portrait based on the user's usage records.

[0169] As a possible implementation method, the electronic device can generate a user portrait that can reflect the user's personalized content, such as one or more interests, preferences, habits, etc., based on the records or application data of the user's use of the application through organization, integration, and refinement.

[0170] As an example, the electronic device can determine the user's favorite music, favorite music style, favorite singer, favorite singer / song era, etc. based on the user's audio and / or video playlists, favorite songs, etc. during the user's use of audio applications; determine the user's hobbies (such as traveling, sports, reading, etc.) based on the frequency of the user's use of various applications; determine the user's personality (such as sunny and cheerful, quiet, etc.) based on the text posted by the user during the use of social applications, etc.

[0171] As an example, after analyzing the user's usage records, the electronic device may determine the user profile as follows: likes to listen to the songs of singer S, likes traveling, likes reading (recently reading Su Dongpo's poems), and tends to live a quiet life.

[0172] S805: The electronic device sends content summaries and user portraits of the multiple images to the server.

[0173] S806: The server constructs prompt information according to the content summaries of the multiple images and the user portrait.

[0174] As a possible implementation, the server may first perform a first process on the content summary and the user portrait, and then perform analysis and reasoning based on the prompt information obtained after the first process, select relevant materials from the material library and determine the image template. For example, the first process may include splicing the content summary and the user portrait based on a preset rule. For another example, the first process may include optimizing the content summary and / or the user portrait and then splicing the content based on a preset rule.

[0175] As an example, the prompt information constructed by the server based on the content summaries of multiple pictures and the user portrait may be as follows:

[0176] "You are a memory image generation assistant. Your task is to assist in creating memory images that meet the user's personalized preferences based on the content summary and user portrait. You can call the material library query tool, retrieve existing filters, background music, physique, copywriting and other materials. The following is the content summary and user portrait: content summary xxxx, user portrait xxxx".

[0177] S807: The server calls the image template model and determines the image template based on the material in the material library according to the prompt information.

[0178] As an example, the server determines that the image template may represent the following multimodal information:

[0179] a) Memory image theme: Good memories of scenic spot B;

[0180] b) Picture sorting: Picture 1 → Picture 3 → Picture 2;

[0181] c) Copywriting: In the sunny March, I took my children to stroll in the gardens of scenic area B in city D of province A, as if I had traveled back to the ancient water town of Jiangnan. We shuttled between the exquisite corridors, admired the rockery and flowing water, and savored the leisurely life of the ancients. The children fed the fish by the pond, and their laughter echoed between the quaint buildings...;

[0182] d) Filter: Ancient style;

[0183] e) Transition effect: alternating between near and far;

[0184] f) Background music: "xxx" by singer S.

[0185] S808: The server sends the image template to the electronic device.

[0186] S809: The electronic device fills the content according to the image template to obtain a recall image.

[0187] Understandably, based on Figure 8 In the process of making a memory image based on a picture, the end-side electronic device can send the content summary obtained according to the original information of the picture and the recognition result, as well as the user portrait used to characterize the user's personalized content, to the cloud-side server, so that the server can automatically analyze and reason, select relevant materials from the material library, and determine the image template used to indicate multimodal information and send it to the electronic device, so that the electronic device can obtain the memory image based on the image template. In this solution, since the original information of the picture in the electronic device and the records or application data of the user during the use of the application are not used as inputs for determining the image template, privacy or security risks can be avoided. In addition, this solution can not only utilize the portability and real-time performance of the end-side electronic device, but also utilize the powerful computing power and rich material resources of the large model in the cloud-side server. There are very few links that require manual participation by the user, or even no manual participation by the user. It can not only simplify user operations, but also obtain memory images with richer content, more natural effects, and more realistic scenes, including user-related story contexts, plots, and timelines, but also improve the overall processing efficiency, while reducing the requirements for the processing performance of the end-side electronic device.

[0188] Moreover, the memory images finally produced by this method can not only generate memory images with rich content, but also the themes, texts, background music, filters, styles, special effects, transition effects and other contents can match the personalized content related to the picture and the personalized content related to the user. Therefore, it is possible to obtain memory images with richer content, more natural effects and more close to the actual scene, which contain user-related story context, plot and timeline, meet the personalized requirements of users, and achieve the memory image generation effect of thousands of faces for thousands of people. For example, Fig. 9 As shown, based on Fig. 9 The pictures 1, 2, 3, ... on the electronic device interface 900 shown in (a) can be obtained. Fig. 9 The memory image is shown in (b) in the figure. Fig. 9 As shown in (b), as the memory image is played, the electronic device displays the good times about Picture 1, Picture 2, and Picture 3 through interface 901, interface 902, and interface 903 respectively, accompanied by the user's favorite music. The memory image is rich in content and closely fits the picture and the user's personalized information, so it can provide the user with a good memory image viewing experience.

[0189] Furthermore, the method can provide a variety of different forms of memory image generation, such as video, slide show or album, so it can meet the diverse needs of users and display a variety of memory images. Fig.10 , Fig.10 Two examples of memory images provided in the embodiments of the present application are shown. Fig.10 As shown in the interface 1001 in the figure, based on the memory image production method provided in the embodiment of the present application, a travel track memoir including multiple memory images (such as memory image 1, memory image 2 and memory image 3) can be obtained; Fig.10 As shown in interface 1002, based on the memory image production method provided in the embodiment of the present application, a memoir of today for each year can be obtained, including multiple memory images (such as memory images of today in 2019, today in 2020, today in 2021, today in 2022, and today last year).

[0190] It should be noted that the above embodiment of the present application only generates a scene of memory images including influencing elements (such as pictures and / or videos, etc.) stored in the electronic device as an example. In practical applications, the method provided by the embodiment of the present application can also be applied to social media, e-commerce, advertising and marketing, etc. For example, a supplier can generate personalized product recommendation videos and advertising copy based on the user's purchase history and / or browsing behavior; for another example, a social platform can generate personalized news push and content recommendations based on the user's interests and / or interaction records.

[0191] It should be understood that the various schemes of the embodiments of the present application can be used in reasonable combination, and the explanations or descriptions of the various terms appearing in the embodiments can be mutually referenced or explained in the various embodiments, without limitation.

[0192] It should also be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0193] It is understandable that, in order to implement the functions of any of the above-mentioned embodiments, the electronic device or server includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0194] The embodiments of the present application can divide the electronic device or server into functional modules. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0195] It should also be understood that each module in the electronic device or server can be implemented in software and / or hardware form, without specific limitation. In other words, the electronic device or cloud server is presented in the form of functional modules. The "module" here can refer to a specific application integrated circuit ASIC, a circuit, a processor and memory that executes one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions.

[0196] In an optional manner, when data transmission is implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is implemented in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disk (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0197] The steps of the method or algorithm described in conjunction with the embodiments of the present application can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in an electronic device or a server. Of course, the processor and the storage medium can also exist as discrete components.

[0198] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

Claims

1. A method for producing a memory image, characterized in that: The method comprises: determining a target image element from among the stored multiple image elements; Obtaining a content summary and a user portrait of the target image element, wherein the user portrait is used to represent the personalized content of the user and is generated based on the user's history of using the electronic device; According to the content summary and the user portrait, an image template is determined based on the materials in the material library.

2. The method according to claim 1, characterized in that The method further comprises: Content is filled according to the image template to obtain a recall image.

3. The method according to claim 1, characterized in that: The method further comprises: Accepting the modification of the image template by the user to obtain the modified image template; Content is filled according to the modified image template to obtain a recall image.

4. The method according to any one of claims 1 to 3, characterized in that The method is applied to electronic equipment.

5. The method according to any one of claims 1 to 3, characterized in that: The step of determining a target image element from among the stored multiple image elements and obtaining a content summary and a user portrait of the target image element is performed by an electronic device; the step of determining an image template based on materials in a material library according to the content summary and the user portrait is performed by a server; and the method further includes: The electronic device sends the content summary and the user portrait to the server; After determining the image template, the server sends the image template to the electronic device.

6. The method according to any one of claims 1 to 5, characterized in that The step of determining a target image element from among the stored plurality of image elements comprises: At least one image element selected by the user from the stored multiple image elements is used as the target image element; or, A plurality of image elements with preset relevance are determined from the plurality of stored image elements as the target image elements, wherein the preset relevance includes any one or more of the following: time relevance, location relevance, season relevance, object relevance, theme relevance, and description relevance.

7. The method according to any one of claims 1 to 6, characterized in that Get the content summary of the target image element, including: The content summary is obtained according to the original information of the target image element and / or the recognition result of the target image element.

8. The method according to any one of claims 1 to 7, characterized in that Obtaining the user portrait includes: The user portrait is obtained based on the records or application data of the user using the application, and the user portrait is used to reflect one or more of the following of the user: interests, preferences, and habits.

9. The method according to any one of claims 1 to 8, characterized in that The material library includes one or more of the following: layout templates, text, pictures, videos, stickers, graffiti, filters, styles, music, and special effects.

10. The method according to any one of claims 1 to 9, characterized in that The image template is used to indicate multimodal information, and the multimodal information includes the form of the image template, the layout position of the target image element, the display size of the image element, and the recall image theme; The image template may be in the form of a video, a slide show, or a picture collection.

11. The method according to claim 10, characterized in that The image template is also used to indicate any one or more of the following: the text and text position of the target image element, the background music name, the background music style, the background music playback duration, the background music playback method, the filter type, style, special effect type, transition effect, stickers and / or graffiti and the corresponding layout position.

12. The method according to any one of claims 2 to 11, characterized in that: The image template is not filled with relevant materials, and the content filling is performed according to the image template to obtain the recall image, including: Relevant materials are edited and target image elements are arranged according to the image template to obtain the recalled image.

13. The method according to any one of claims 2 to 11, characterized in that: The image template has been filled with relevant materials, and the content filling is performed according to the image template to obtain the recall image, including: The target image elements are arranged according to the image template to obtain the recalled image.

14. A method for producing a memory image, characterized in that: Applied to a server, the method comprises: Receiving a content summary and a user portrait of a target image element from an electronic device, wherein the user portrait is used to represent personalized content of a user and is generated based on a history record of the user using the electronic device; Determine an image template based on the material in the material library according to the content summary and the user portrait; The image template is sent to the electronic device.

15. The method according to claim 14, characterized in that The material library includes one or more of the following: layout templates, text, pictures, videos, stickers, graffiti, filters, styles, music, and special effects.

16. The method according to claim 14 or 15, characterized in that The image template is used to indicate multimodal information, and the multimodal information includes the form of the image template, the layout position of the target image element, the display size of the image element, and the recall image theme; The image template may be in the form of a video, a slide show, or a picture collection.

17. The method according to claim 16, characterized in that The image template is also used to indicate any one or more of the following: the text and text position of the target image element, the background music name, the background music style, the background music playback duration, the background music playback method, the filter type, style, special effect type, transition effect, stickers and / or graffiti and the corresponding layout position.

18. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer program instructions; A processor, configured to execute the computer program instructions to support the electronic device to implement the method as described in any one of claims 1-13.

19. A server, characterized in that: The server comprises: a memory for storing computer program instructions; A processor, configured to execute the computer program instructions to support the electronic device to implement the method as described in any one of claims 14-17.

20. A memory image production system, characterized in that: The system comprises: The electronic device as claimed in claim 18, and the server as claimed in claim 19.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processing circuit, the method according to any one of claims 1 to 13 or claims 14 to 17 is implemented.

22. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 13 or claims 14 to 17.

Images