Image processing method and electronic device

Abstract

Embodiments of the present application relate to the technical field of terminal devices, and provide an image processing method and an electronic device. The method comprises: displaying an interactive UI control on an image on the basis of a user operation, so that the function and service of the UI control can be embedded in the image for use by the user.

Description

Image processing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 11, 2023, with application number 202311176694.X and application name “Image Processing Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the technical field of terminal devices, and in particular to an image processing method and electronic device. Background Art

[0003] With the development of technology, electronic devices can provide UI controls such as service cards to facilitate users to use corresponding functions or services of applications or electronic device systems.

[0004] Taking a mobile phone as an example, in some scenarios, users may wish to add a UI control (e.g., a service card) supported by the mobile phone to a picture for display. In the prior art, users can overlay, crop, or erase a screenshot of a service card and the picture to which the card is to be added to display the service card.

[0005] However, in the prior art, the pictures with service cards added thereto are merely static images of the service cards, and are unable to interact to provide the functions of the service cards.

[0006] Summary of the Invention

[0007] To solve the above technical problems, the present application provides an image processing method and electronic device. In this method, an interactive UI control can be displayed on an image, so that the functions and services of the UI control can be embedded in the image for user use.

[0008] In a first aspect, embodiments of the present application provide an image processing method, applied to an electronic device. The method comprises: displaying a target image in response to a first user operation received on a first image, wherein the target image is a first image displaying a target user interface (UI) control; and outputting an interaction result related to the target UI control in response to a second user operation received on the target UI control.

[0009] The first image may be an original image without any UI control added thereto, or may be an image after a first UI control in a picture format is added to the original image, etc., and there is no limitation here.

[0010] In one example, the first user operation may be an operation of adding an interactive UI control to the original image. The operation may be any form of screen touch input or a physical button press input, etc., which is not limited here.

[0011] In another example, the first image is an image with a UI control in picture format added. Then, when the user performs an operation to request the display of the first image (for example, the first user operation is an opening operation on the first image), the electronic device can execute the above method to instantiate the target UI control on the first image.

[0012] In yet another example, the first user operation may also be a photographing operation, so as to instantiate and display the target UI control on the first photographed image.

[0013] In addition, in some examples, the first user operation may also include a selection operation on a target UI control to select the UI control to be added to the first image this time.

[0014] In addition, the above-mentioned target UI control in the first aspect may be a UI control added by the first user operation instruction, and / or may be a UI control that the electronic device automatically determines to match with the first image in combination with the first image.

[0015] The target UI control can be any interactive UI control. For example, the control type of the target UI control can be a service card (also called a widget in some scenarios). Service cards are also divided into many types, such as music cards, map cards, navigation cards, video cards, sports cards, etc. The card types (i.e., control types) of various service cards are also different. The subsequent implementation method uses the UI control being a service card as an example to illustrate the method of this application.

[0016] In addition, the target UI control can also be a notification message, a live window, a floating control (such as a floating window, a floating ball, etc.), an interactive button (such as a button for making a call to a contact), a text box (such as a search box), an application interface (such as a chat interface, etc., without limitation), etc.

[0017] In an embodiment of the present application, an electronic device may display a target UI control on a first image in response to a user operation, and the target UI control is an interactive UI control. When a user operates the target UI control, the electronic device may output an interactive result related to the target UI control in response to the user's operation on the target UI control. This makes the target UI control displayed on the target image not a static image, but an interactive UI control. By interacting with the UI control displayed on the target image, the user can use the functions and services provided by the UI control to expand the usage scenarios of the UI control. In this way, the UI control embedded in the target image can also provide functions and services related to the UI control for user interaction.

[0018] Among them, the way of outputting the interaction result related to the target UI control may include but is not limited to: displaying the application interface, refreshing the display information of the UI control, outputting audio, etc.

[0019] For example, if the target UI control is a music card, the second user operation may be clicking the play button on the music card. In this way, when the electronic device outputs the interaction result, it can refresh the triangular icon of the play button of the music card displayed on the first image to a double vertical line to refresh the display information of the music card. In addition, it can also output the audio data of the song displayed on the music card to play the song with the title displayed on the music card.

[0020] For another example, if the target UI control is a search box, then after the user enters a search object in the search box through a second user operation and triggers a search operation, the electronic device may display an application interface of a search engine application when outputting interaction results. The application interface is a search results interface, and the search results in the search results interface are the search results of the search object obtained by the search engine application, which provides the aforementioned search box.

[0021] According to the first aspect, the target UI control is related to a music application running in the background when the first user operation is received.

[0022] For example, the target UI control is a music control. The music control may be related to a music application running in the background when the electronic device receives the first user operation.

[0023] For example, when a user selects a picture in a gallery application through a first user operation to request that a service card be added to it (or when the user uses a camera application to take a picture through a first user operation), and a music application is running in the background of the electronic device (for example, music is being played using the music application), the electronic device can add music controls related to the music application to the picture.

[0024] For example, the music control can be a music card, and the card information displayed on the music card may include information about the song currently playing in the music application. Of course, the music control can also be other types of interactive music controls, such as a music control that includes a play / pause button for the music and the name of the music, or a music control that is a floating window on the music playback interface of the music application. The music control of the present application can be any interactive UI control related to the above-mentioned music application, such as a service card, a floating control, an application interface, etc.

[0025] Optionally, the control information displayed by the music control may be related to the music application.

[0026] In an embodiment of the present application, the electronic device can add interactive music controls to the first image in combination with the music application running in the background when the user adds the UI control to the first image, so that the target UI control displayed on the target image can be related to the music scenario when the user adds the UI control.

[0027] According to the first aspect, or any implementation of the first aspect above, the target UI control is related to the first image.

[0028] In some implementations, the target UI control may have target control metadata to describe the target UI control.

[0029] The target control metadata may include but is not limited to: control type, control size, control display position, control display information, and control image parameters.

[0030] The control display information is used to describe the content displayed on the control, and the control display information may include but is not limited to at least one of: a button, an icon, an image, and text.

[0031] The image parameters of the control may include but are not limited to at least one of perspective parameters, color, hue, transparency, and lighting characteristics.

[0032] At least one of the above-mentioned metadata in the target UI control may be related to the first image. In other words, the electronic device may determine at least one attribute of the target UI control based on the first image, wherein the above-mentioned target control metadata may also be described as an attribute of the target UI control.

[0033] In some implementations, the control type of the target UI control can be related to the image content of the first image.

[0034] The image content may be the image semantics of the first image, or the scene represented by the first image, etc.

[0035] For example, if the image content of the first image is a building, then its semantics is the building, and the target UI control can be a navigation card or a map card.

[0036] For another example, if the image content of the first image is a picture of a person running in a gym, then the scene indicated by the first image is a sports scene, and the target UI control may be a sports card.

[0037] In some implementations, the control display position of the target UI control may be related to the layout between the foreground elements and the background elements of the first image. For example, the display position of the target UI control on the first image is a specific position between the foreground elements and the background elements.

[0038] In some implementations, the control size of the target UI control may be related to the size, depth of field, etc. of image elements (foreground elements and / or background elements) in the first image.

[0039] For example, if the image content of the first image is a user watching a movie through a projector screen, then the target UI control added to the first image can be a video card. In addition, the size of the video card can be the same as the size of the screen in the first image.

[0040] In some implementations, the control display information of the target UI control may be related to the shooting time, shooting location, image content, etc. of the first image.

[0041] For example, if the target UI control is a weather card, the card information displayed by the weather card may be weather information related to the shooting time and shooting location of the first image.

[0042] In some implementations, the image parameters of the target UI control may be related to information such as the color, hue, transparency, lighting characteristics, perspective parameters, etc. of the foreground elements and / or background elements of the target area in the first image where the target control is to be displayed.

[0043] In an embodiment of the present application, the electronic device can display a target UI control related to the first image on the first image, so that the first image can match at least one of the control type, control size, control display position, control display information, and image parameters of the target UI control. In this way, the UI control added to the picture can be better matched with the picture, and the image fusion and picture harmony between the target UI control and the first image are higher.

[0044] According to the first aspect, or any implementation of the first aspect above, the control type of the target UI control is related to the image content of the first image.

[0045] For example, if the first image includes a building, the electronic device may add a location-related service card, such as a map card or a navigation card, to the first image. Optionally, the map information displayed in the map card may be map information about the building's location, and the navigation card may be navigation information for navigating to the building's address.

[0046] For another example, the image content of the first image is a picture of a person running in a gym, and the electronic device can add a sports card to the first image.

[0047] For example, if the image content of the first image is a user holding a microphone and singing, the electronic device can add a music card to the first image.

[0048] For example, if the image content of the first image is a scene of a person making a phone call, the electronic device may add a contact card, or display a target UI control such as a button for making a call to a specific contact on the first image.

[0049] In this way, the present application can combine the image content of the first image to add a corresponding type of UI control that better matches the image content to the first image, so that the image content of the first image displaying the target UI control is more compatible with the control type of the added target UI control.

[0050] According to the first aspect, or any implementation of the first aspect above, the control information displayed by the target UI control is related to the image content of the first image.

[0051] In some implementations, the control display information of the target UI control (also expressed as displayed control information) may be related to the shooting time, shooting location, image content, etc. of the first image.

[0052] For example, if the target UI control is a weather card, the card information displayed by the weather card may be weather information related to the shooting time and shooting location of the first image.

[0053] In this way, the control display information of the interactive target UI control added by the electronic device to the first image can match the image content of the first image, so that the first image and the target UI control have a higher degree of semantic integration and matching.

[0054] According to the first aspect, or any implementation of the first aspect above, the perspective parameter of the target UI control is the same as the perspective parameter of the target image element corresponding to the display position of the target UI control in the first image.

[0055] The aforementioned display position is the position in the first image where the target UI control is to be displayed (also referred to as the first area). If the target image pixels (foreground elements and / or background elements, depending on the display position of the target UI control) in the first area of ​​the first image where the target UI control is to be added have a tilt angle, for example, if a curtain element in the first image is not photographed perpendicular to its surface, the curtain element will have a tilt angle when displayed in the first image. For another example, if the glass of a building displayed in the first image is curved, the glass displayed in the first image will always have a tilt angle regardless of the viewing angle from which the building is photographed. In such cases, the target image elements in the first image (e.g., building glass elements, curtain elements, etc.) have perspective parameters. When the electronic device adds a UI control to the area of ​​the first image where the target image element is located, the UI control can overlap the target image element. To ensure that the tilt angle of the target image element is consistent with that of the UI control, thereby achieving a better fit between the two, the electronic device can set the perspective parameters of the target UI control added to the first image to be the same as those of the target image element. The perspective parameter may include a perspective direction and / or a perspective angle.

[0056] In this way, when the target image element of the target UI control displayed on the first image has a tilt angle, the target UI control added to the first image in the present application is also adjusted through corresponding perspective parameters, so that the tilt angle of the target UI control added to the first image can be consistent with the tilt angle of the above-mentioned target image element. In this way, the target UI control displayed on the first image can be more closely aligned with the target image element at the corresponding display position, and the target UI control can be more integrated with the image elements around the target image element.

[0057] According to the first aspect, or any implementation manner of the first aspect above, displaying the target image includes: displaying a second image, wherein the second image is the first image including a first UI element, wherein the first UI element is a first UI control in a picture format; displaying a target UI control related to the first UI control on the second image to display the target image, wherein the first UI element is covered by the target UI control.

[0058] In this implementation, the first image is an original image without UI elements added to it. Then, when the user requests to add UI elements to it through a first user operation, the electronic device of the present application may first add a first UI control in a picture format to the first image to obtain a second image; then, when the electronic device displays the target UI control on the second image, it may be triggered to display the target UI control on the second image in response to another user operation received, or it may be automatically displayed in the second image after the second image is displayed. The corresponding position of the first UI control in a picture format (because it is a picture format, it is not interactive) is automatically displayed in the second image to instantiate the display target UI control, so that the first UI control in the picture format can be covered by the target UI control. Of course, the display range of the target UI control on the second image may also exceed the display range of the first UI control in the picture format in the second image, so as to achieve the effect that the target UI control covers the first UI control.

[0059] In some embodiments, the display area corresponding to the target UI control on the first image includes the display area corresponding to the first UI element on the first image to ensure that the interactive target UI control can cover the first UI element in a static image format in the first image.

[0060] In addition, the instantiated target UI control can be of the same control type as the first UI control, or the target UI control and the first UI control have the same control display information, or the target UI control and the first UI control are the same UI control. There is no restriction here. As long as the target UI control is related to the first UI control, the correlation between the two can be the correlation of any type of metadata in the target control metadata mentioned in the above implementation method.

[0061] In one example, the electronic device may generate a target UI control (also referred to as an instantiated target UI control) based on the first UI element in the first image, and display the target UI control at a corresponding position in the second image.

[0062] For example, first control metadata about a first UI control of a first UI element may be carried in metadata of a first image, and the electronic device may generate a target UI control related to the first UI control based on the first control metadata.

[0063] For another example, the electronic device may perform image recognition on the second image to determine first control metadata of a first UI control in a picture format embedded in the second image, and generate a target UI control related to the first UI control in combination with the first control metadata.

[0064] In addition, in some embodiments, when the electronic device generates a second image based on the first image, or generates the above-mentioned target image based on the first image or the second image, the first image can be divided into a foreground image where the foreground elements are located and a background image where the background elements are located, wherein the foreground image can be one or more images and the background image can be one or more images.

[0065] Among them, the foreground image is an image that only includes the foreground element (the area other than the foreground element is a transparent area), and the background image is an image that only includes the background element obtained by segmenting from the first image. The sizes of the foreground image and the background image are the same as those of the first image, the second image, and the target image.

[0066] In some embodiments, the first UI control in the above-mentioned picture format, the foreground image obtained by segmenting the first image, and the background image obtained by segmenting the first image can be sequentially used as three layers from top to bottom to synthesize the second image.

[0067] Alternatively, in some embodiments, the foreground image obtained by segmenting the first image, the first UI control in the picture format, and the background image obtained by segmenting the first image can be sequentially used as three layers from top to bottom to synthesize a second image. Optionally, in the second image, the first UI control in the picture format can be partially obscured by foreground elements in the foreground image.

[0068] In some embodiments, when the electronic device directly uses the first image to generate the target image without generating the second image, the foreground image segmented from the first image, the target UI control, and the background image segmented from the first image can be sequentially used as three layers from top to bottom to synthesize the target image. Alternatively, the target UI control can also be used as the topmost image.

[0069] In some embodiments, when the electronic device generates a target image based on the second image, the layer where the target UI control is located may be located above the layer where the first UI control in the image format is located, so as to achieve the target UI control blocking the first UI control.

[0070] In this implementation, the electronic device may embed a first UI control in a picture format on a first image to display a second image with the first UI control in a picture format, and then, based on the display position of the first UI control in the picture format in the second image, instantiate a target UI control (for example, create a service card at the display position) to obtain a target image with an interactive target UI control. For example, the electronic device may save the second image, and subsequently, when the second image is displayed, instantiate the target UI control at the corresponding display position to make the target UI control displayed on the target image interactive.

[0071] According to the first aspect, or any implementation of the first aspect above, the control information displayed by the first UI control and the target UI control (ie, the control display information mentioned in the control metadata) is the same.

[0072] In some embodiments, the first UI control and the target UI control can be the same control. In this way, the instantiation process of the target UI control, that is, the process of the image displayed by the electronic device changing from the second image to the target image, is indistinguishable to the user by the naked eye. That is, the first UI control in picture format in the second image and the interactive target UI control in the target image have no difference in display effect, but the first UI control in the second image is not interactive.

[0073] Alternatively, in other embodiments, although the first UI control and the target UI control display the same control information, the first UI control and the target UI control may be different types of controls; or, the first UI control and the target UI control are the same type of controls and display the same control information, but the two may differ in at least one aspect of the control border, shadow, hue, size, etc., so that the user can distinguish that the second image displays a non-interactive UI control, while the target image displays an interactive UI control.

[0074] According to the first aspect, or any implementation of the first aspect above, the size of the target UI control may be slightly larger than or equal to the size of the first UI element.

[0075] According to the first aspect, or any implementation of the first aspect above, the control information displayed by the first UI control and the target UI control may differ to some extent. For example, the card information of a weather card in image format is weather information related to the shooting time and location of the first image. However, the card information displayed by the instantiated weather card is card information related to the current time and location of the mobile phone.

[0076] According to the first aspect, or any implementation of the first aspect above, the method further includes: storing first control metadata of the first UI control in association with the second image; and generating the target UI control based on the first control metadata.

[0077] The associated storage method may be, for example, storing the first control metadata in the metadata of the second image, or associating the first control metadata with the name of the second image.

[0078] In addition, the process of the electronic device generating the target UI control based on the first control metadata can be understood as a process of instantiating (ie, creating and displaying) the target UI control on the second image to obtain the target image.

[0079] The first control metadata can be identical to the target control metadata, i.e., the target UI control and the first UI control are the same control. Alternatively, the first control metadata and the target control metadata can be of the same control type, but there can be at least one metadata difference between the two (i.e., the control property mentioned above), such as a different control size, where the target UI control is larger than the first UI control.

[0080] In this embodiment, the electronic device can use the first control metadata associated with the second image containing the first UI control in picture format to generate a target UI control to obtain a target image. In this way, the target UI control instantiated and displayed on the target image is related to the first UI control, and the instantiated interactive target UI control meets the user's requirements for the UI control added to the first image.

[0081] According to the first aspect, or any implementation of the first aspect above, the method further includes: refreshing the first control metadata as the target control metadata of the target UI control.

[0082] For example, if there is a certain difference between the target control metadata and the first control metadata, in order to ensure that when the second image is shared with other electronic devices, the interactive UI control displayed on the first image by other electronic devices using the method of the present application is the same as the target UI control, the electronic device can refresh or replace the locally stored first control metadata with the target control metadata.

[0083] According to the first aspect, or any implementation of the first aspect above, the method further includes: in response to a received display operation on the second image, displaying a target UI control related to the first UI control on the second image to display the target image.

[0084] In this embodiment, after the user adds a first UI control in a picture format to a first image through a first user operation to obtain a second image, the electronic device can save the second image. Then when the user operates to request the display of the second image, the electronic device can respond to the operation to instantiate the target UI control on the second image to display the target image. In this way, when the user needs to open a second image displaying a first UI element, the electronic device can, for example, instantiate the target UI control (for example, the target UI control is the same as the first UI control) at the display position of the first UI element in the second image to display the target image. In this way, although the electronic device saves a second image with a non-interactive UI control (i.e., the first UI element), when the second image needs to be displayed, the electronic device can display an interactive target UI control on the second image to facilitate the user to interact with the target UI control in the target image.

[0085] According to the first aspect, or any implementation of the first aspect above, the method further includes: in response to a received sharing operation on the second image, sending the second image and the first control metadata associated with the second image.

[0086] In this embodiment, when an electronic device needs to share a second image with another electronic device, the first control metadata associated with the second image can be synchronously shared with the other electronic device. For example, if the first control metadata has been refreshed as the target control metadata in the above implementation, the electronic device can send the target control metadata to the other electronic device. After the other electronic device receives the second image and its first control metadata, when the other electronic device displays the second image, it can instantiate the target UI control according to the first control metadata to display the target image on the other electronic device, thereby realizing the sharing of the image with the interactive UI control.

[0087] According to the first aspect, or any implementation of the first aspect above, displaying the target image includes: generating a target application; and displaying the target image in response to a received startup operation for the target application.

[0088] In an embodiment, there is no need to first add a first UI element in a picture format to the first image. Instead, when the user requests to add a UI control to the first image through a first user operation, the target UI control can be instantiated on the first image to generate a target application. The target application may include the first image instantiated with the target UI control, that is, the target image. Then, each time the user starts the target application, the electronic device can display the target image stored in the target application. Compared with the above-mentioned implementation method of generating a second image and then obtaining the target image based on the second image, the method of this embodiment is more efficient in displaying an image with a UI control.

[0089] In a second aspect, embodiments of the present application provide an electronic device. The electronic device includes: a first display module configured to display a target image in response to a first user operation received on a first image, wherein the target image is a first image displaying a target user interface (UI) control; and an output module configured to output an interaction result related to the target UI control in response to a second user operation received on the target UI control.

[0090] According to the second aspect, the target UI control is a music control, and the music control is related to a music application running in the background when the first user operation is received.

[0091] According to the second aspect, or any implementation of the second aspect, the target UI control is related to the first image.

[0092] According to the second aspect, or any implementation of the second aspect above, the control type of the target UI control is related to the image content of the first image.

[0093] According to the second aspect, or any implementation of the second aspect above, the control information displayed by the target UI control is related to the image content of the first image.

[0094] According to the second aspect, or any implementation of the second aspect above, the perspective parameter of the target UI control is the same as the perspective parameter of the target image element corresponding to the display position of the target UI control in the first image.

[0095] According to the second aspect, or any implementation method of the above second aspect, the first display module is specifically used to: display a second image, wherein the second image is the first image including a first UI element, wherein the first UI element is a first UI control in a picture format; display a target UI control related to the first UI control on the second image to display a target image, wherein the first UI element is covered by the target UI control.

[0096] According to the second aspect, or any implementation of the second aspect, the first UI control and the target UI control each display the same control information.

[0097] According to the second aspect, or any implementation of the second aspect above, the electronic device also includes: a storage module for storing the first control metadata of the first UI control in association with the second image; and a generation module for generating the target UI control based on the first control metadata.

[0098] According to the second aspect, or any implementation of the second aspect above, the electronic device further includes: a refresh module, configured to refresh the first control metadata to the target control metadata of the target UI control.

[0099] According to the second aspect, or any implementation of the second aspect above, the electronic device also includes: a second display module, used to display a target UI control related to the first UI control on the second image in response to a received display operation on the second image, so as to display the target image.

[0100] According to the second aspect, or any implementation of the second aspect above, the electronic device further includes: a sending module for sending the second image and the first control metadata associated with the second image in response to a received sharing operation on the second image.

[0101] According to the second aspect, or any implementation of the second aspect above, the first display module is specifically configured to: generate a target application; and display a target image in response to a received startup operation for the target application.

[0102] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0103] In a third aspect, an embodiment of the present application provides an electronic device. The electronic device includes: a memory and a processor, the memory and the processor being coupled; the memory storing program instructions, which, when executed by the processor, cause the electronic device to perform the method according to the first aspect and any one of the implementations of the first aspect.

[0104] The technical effects corresponding to the third aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.

[0105] In a fourth aspect, embodiments of the present application provide a computer-readable medium for storing a computer program, which, when executed on an electronic device, causes the electronic device to execute the method according to the first aspect and any one of the embodiments of the first aspect.

[0106] The technical effects corresponding to the fourth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.

[0107] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer is caused to execute the method in the first aspect and any one of the implementations of the first aspect.

[0108] The technical effects corresponding to the fifth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.

[0109] In a sixth aspect, embodiments of the present application provide a chip. The chip includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from a memory of a first electronic device and send the signals to the processors, the signals including computer instructions stored in the memory; when the processors execute the computer instructions, the first electronic device executes the method according to the first aspect and any one of the implementations of the first aspect.

[0110] The technical effects corresponding to the sixth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] FIG1 is a schematic diagram showing a structure of an electronic device;

[0112] FIG2 is a schematic diagram illustrating a software structure of an electronic device;

[0113] FIG3 a is a schematic diagram showing one of the exemplary processing procedures;

[0114] FIG3 b is a schematic diagram showing one of the exemplary processing procedures;

[0115] FIG3 c is a schematic diagram showing one of the exemplary sharing processes;

[0116] FIG4 is a schematic diagram showing one of the exemplary processing procedures;

[0117] FIG5a is a schematic diagram of one of the exemplary application scenarios;

[0118] FIG5 b is a schematic diagram of one of the exemplary application scenarios;

[0119] FIG5c is a schematic diagram of one of the exemplary application scenarios;

[0120] FIG5 d is a schematic diagram of one of the exemplary application scenarios;

[0121] FIG6 a is a schematic diagram of one of the exemplary application scenarios;

[0122] FIG6 b is a schematic diagram of one of the exemplary application scenarios;

[0123] FIG6c is a schematic diagram of one of the exemplary application scenarios;

[0124] FIG6 d is a schematic diagram of one of the exemplary application scenarios;

[0125] FIG7a is a schematic diagram of one of the exemplary application scenarios;

[0126] FIG7 b is a schematic diagram of one of the exemplary application scenarios;

[0127] FIG7c is a schematic diagram of one of the exemplary application scenarios;

[0128] FIG7 d is a schematic diagram of one of the exemplary application scenarios;

[0129] FIG8a is a schematic diagram of one of the exemplary application scenarios;

[0130] FIG8 b is a schematic diagram of one of the exemplary application scenarios;

[0131] FIG9 is a schematic diagram of one of the exemplary application scenarios;

[0132] FIG10a is a schematic diagram of one of the exemplary application scenarios;

[0133] FIG10 b is a schematic diagram of one of the exemplary application scenarios;

[0134] FIG11 is a schematic diagram of the structure of the device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0135] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0136] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0137] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0138] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0139] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0140] FIG1 shows a schematic diagram of the structure of an electronic device 100. It should be understood that the electronic device 100 shown in FIG1 is only an example of an electronic device. Alternatively, the electronic device 100 may be a terminal, which may also be referred to as a terminal device. The terminal may be a cellular phone, a tablet computer (pad), a wearable device, an Internet of Things device, an in-vehicle device, or other device with a display screen or an external display screen, which is not limited in this application. It should be noted that the electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in FIG1 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0141] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0142] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0143] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0144] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0145] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0146] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0147] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0148] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0149] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0150] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0151] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0152] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0153] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0154] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0155] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0156] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0157] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0158] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0159] The camera 193 is used to capture still images or videos. The object is projected onto the photosensitive element through the lens to generate an optical image.

[0160] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0161] The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 enables the electronic device 100 to execute the method in the embodiment of the present application by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0162] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0163] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0164] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0165] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0166] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0167] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting it into or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0168] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0169] FIG2 is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application.

[0170] The layered architecture of electronic device 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided into four layers: from top to bottom, the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0171] The application layer can include a series of application packages.

[0172] As shown in FIG2 , the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

[0173] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0174] As shown in FIG2 , the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, a user interface (UI) framework module, and the like.

[0175] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0176] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0177] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0178] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0179] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0180] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0181] The gallery application can call the UI framework module to convert the UI control into a UI control in a picture format, and add the UI control in the picture format to the picture in the selected gallery application.

[0182] In addition, the gallery application can also call the UI framework module to add the location of the UI control in the picture format in the above picture when displaying the picture, and instantiate the corresponding UI control so that the UI control displayed on the picture is an interactive UI control rather than a static image. The specific processing process will be described in detail in the following embodiments.

[0183] Among them, the UI control can be any interactive UI control, such as a service card, a notification message, a live window, a floating control (such as a floating window, a floating ball, etc.), an interactive button (such as a button for making a call to a contact), a text box (such as a search box, etc.), an application interface (such as a chat interface, etc., without limitation).

[0184] Taking the UI control as a service card as an example, the instantiation process of the UI control is explained. Instantiating a service card at a certain location means creating a service card at that location, making the card displayed at that location interactive.

[0185] The system library and runtime layer include the system library and the Android runtime. The system library can include multiple functional modules, such as the surface manager, media libraries, 3D graphics processing libraries (such as OpenGL ES), and 2D graphics engines (such as SGL). The 3D graphics library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing. The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one part contains the function calls required by the Java language, and the other part is the Android core library. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0186] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0187] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0188] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0189] A 2D graphics engine is a drawing engine for 2D drawings.

[0190] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0191] It is understood that the components included in the system framework layer, system library, and runtime layer shown in FIG2 do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or split certain components, or arrange the components differently.

[0192] Currently, with the development of technology, electronic devices can provide service cards to facilitate users to use corresponding functions or services of applications or electronic device systems.

[0193] Taking a mobile phone as an example, in some scenarios, a user may wish to add a service card provided by the mobile phone to a picture for display. In the prior art, the user can overlay, crop, or erase a screenshot of the service card and the picture to which the card is to be added, thereby displaying the service card in the picture.

[0194] However, in the prior art, the pictures with service cards added thereto are merely static images of the service cards, and are not interactive service cards.

[0195] To this end, the present application provides an image processing method and electronic device, which can add UI controls to a picture and process the UI controls in the picture into interactive UI controls.

[0196] Among them, the UI control can be any interactive UI control, such as a service card (also called a widget in some scenarios), a notification message, a live window, a floating control (such as a floating window, a floating ball, etc.), an interactive button (such as a button for calling a contact), a text box (such as a search box), an application interface (such as a chat interface, etc., without limitation), etc.

[0197] Taking the UI control as a service card as an example, in an example 1, the electronic device can call the UI framework module through the gallery application shown in Figure 2 to execute the image processing method, thereby embedding the service card into the selected picture in the form of an image, and when displaying the picture, the position of the service card can be displayed in the selected picture to instantiate the service card, so that the service card displayed in the picture can be interactive.

[0198] It should be understood that the above embodiment uses the gallery application as an example to illustrate the method of the present application. However, the application implemented by the method of the present application is not limited to the gallery application, but can also be other applications that can display pictures. The method is similar and will not be repeated here.

[0199] Taking the UI control as a service card as an example, in Example 2, the electronic device can instantiate a service card on the selected image to generate a target application, which can be added to the application layer shown in Figure 2. The target application can include a target image with an interactive service card. The electronic device can launch the target application to display the target image with the interactive service card and implement the interaction process with the service card.

[0200] Below, taking the electronic device as a mobile phone as an example, combined with application scenarios 1 to 4, the implementation process of the method in the above example 1 of the present application is explained. When the electronic device is another type of electronic device, or when the UI control is another type of interactive UI control, the principle of the implementation process of the method of the present application is the same or similar, and will not be repeated here.

[0201] Example 1

[0202] Please refer to Figures 3a to 3c, and take the mobile phone's gallery application to call the UI framework module as an example to illustrate the implementation process of the method of the present application.

[0203] 3a mainly describes the process in which the mobile phone gallery application can embed the service card into the selected picture in the form of an image by calling the UI framework module.

[0204] FIG3 b mainly describes the process of displaying the position of the service card in the selected picture by the mobile phone and instantiating the service card.

[0205] FIG3 c mainly describes the process of a mobile phone sharing a picture with an image embedded with a service card.

[0206] FIG5 a to FIG5 d and FIG6 a to FIG6 d are schematic diagrams of an application scenario 1 for implementing the method shown in FIG3 a to FIG3 c .

[0207] The method of the present application will be described below with reference to FIG. 3 a to FIG. 3 c , FIG. 5 a to FIG. 5 d , and FIG. 6 a to FIG. 6 d .

[0208] Referring to Figure 3a, the process may include the following steps:

[0209] S100a: The mobile phone determines a first image.

[0210] The first image is an image of the service card to be added, and the first image may be the selected picture mentioned in the above embodiment.

[0211] In some embodiments, the first image can be the current image taken by the user using the camera application of the mobile phone, or it can be an image selected by the user in the local picture (such as the gallery application), or it can be a screenshot image or an image shared by other users, or a frame image in a video, etc. There is no restriction on the source of the first image here.

[0212] In addition, the present application does not impose any restrictions on the display position of the first image. For example, the first image may also be an image dragged by the user to the transfer station.

[0213] S100b, the mobile phone determines the first service card.

[0214] The first service card may be a service card created through user operation, or a service card selected by the user from among those already created, or a service card recommended by the mobile phone based on the first image and / or based on the real-time scenario.

[0215] For example, if the first image includes a building, the mobile phone may recommend a map card that matches the location of the building, and display the address information of the building in the map card.

[0216] For another example, if the first image includes a projection screen, the mobile phone may recommend a video card that matches the image displayed on the projection screen. For example, the card information displayed by the video card may include a video image that matches the image displayed on the projection screen in the first image.

[0217] For another example, when the mobile phone captures the first image, the real-time scene of the mobile phone is that the mobile phone is playing music. The mobile phone can recommend music cards based on the real-time scene (playing music).

[0218] In some embodiments, the card content of the first service card may be related to the first image.

[0219] For example, the card information may be related to the metadata of the first image (such as at least one item such as the shooting location, shooting time, etc.). In the above-mentioned example of the recommended map card, the card information displayed by the recommended map card may be the map information of the shooting location of the first image.

[0220] For another example, the card information may be related to the image frame of the first image. For example, if the first image includes a picture of a person walking, the first service card determined by the mobile phone may be a sports card, and the card information displayed in the sports card may include information about the number of steps related to the picture.

[0221] In this way, the card information and card type of the service card embedded in the image can match the scene or semantics expressed by the image.

[0222] Among them, this application does not limit the execution order of S100a and S100b. The two can be executed in parallel or serially. The specific execution order may vary based on different application scenarios and is not limited here.

[0223] Application Scenario 1

[0224] The following takes application scenario 1 as an example to illustrate the implementation process of the above S100a and S100b shown in FIG3a.

[0225] In application scenario 1, please refer to Figure 5a. As shown in Figure 5a(1), the display interface 201 of the mobile phone is the application interface of the gallery application. The display interface 201 may include multiple controls, which may include but are not limited to: picture 202, control 203, control 207, etc.

[0226] The picture 202 is a picture stored in the gallery application; the control 207 is used to share the picture 202 .

[0227] As shown in Figure 5a(1), the user clicks on the control 203 in the display interface 201, and the mobile phone can respond to the user operation by switching the display interface to the display interface 201 shown in Figure 5a(2). Specifically, the mobile phone can respond to the user operation by displaying the control 204 in the display interface 201. The control 204 may include a rename option, an add card option 2041, a rotation option, etc.

[0228] The rename option is used to rename the image 202 , and the rotate option is used to rotate the image 202 .

[0229] As shown in Figure 5a(2), when the user clicks on the control 2041, the mobile phone can respond to the user operation by switching the display interface to the display interface 300 shown in Figure 5a(3). As shown in Figure 5a(3), the display interface 300 may include one or more controls, which may include but are not limited to: a search control 301, a map card 302, a weather card 303, a control 304 for setting a combination card, a control 305 for setting a versatile card, a control 306 for adding a music card of the music application, a control 307 for adding a video card of the Huawei video application, a control 308 for adding a sports card of the sports health application, etc.

[0230] The search control 301 is used to search for service cards that can be provided by the mobile phone.

[0231] In addition, the display interface 300 shown in Figure 5a (3) also includes the text "Map (9)" located below the map card 302, which is used to indicate that the mobile phone provides 9 types of map cards related to map applications, and at least one of the display style, display content, etc. of each map card may be different.

[0232] In addition, the display interface 300 shown in Figure 5a (3) also includes the text "Weather (3)" located below the weather card 303, which is used to indicate that the mobile phone provides three weather cards related to weather applications, and at least one of the display style, display content, etc. of each weather card may be different.

[0233] If the user wishes to add a weather card to the image 202 shown in FIG5a(2), then as shown in FIG5a(3), the user clicks on the weather card 303, and the mobile phone can respond to the user operation by switching the display interface of the mobile phone to the display interface 401 shown in FIG5a(4). The display interface 401 may include three types of weather cards for preview, namely weather card 303, weather card 402, and weather card 403. The three types of weather cards are the three types of weather cards for the weather application indicated by the text "Weather (3)" shown in FIG5a(3). And the weather card 303 displayed on the top layer of the three weather cards previewed by the mobile phone indicates the currently selected weather card.

[0234] In addition, as shown in FIG5a(4), the display interface 401 also displays the text “Real-time Weather” to indicate that the card information in each of the weather cards 303, 402, and 403 may include: weather information provided by the weather application that matches the system time and location of the mobile phone.

[0235] For example, let's take weather card 303. The phone's system time is 8:00 AM on July 1, 2023, and the phone is located in Beijing. The temperature information (current temperature, maximum temperature, minimum temperature, etc.) and the weather alert "Thunderbolt and Heavy Rain Warning" provided by weather card 303 are the weather app's forecast for Beijing on July 1, 2023: the current temperature (27°C), maximum temperature (here: 30°C), and minimum temperature (here: 23°C). The same applies to weather cards 402 and 403, so we won't go into detail here.

[0236] In addition, as shown in FIG. 5 a ( 4 ), the display interface 401 may further include a control 404 , a control 405 , and a control 406 .

[0237] Among them, the control 405 can be used to add the selected weather card 303 to the desktop interface display.

[0238] For example, in the embodiment of Figure 5a, the shooting location of the picture 202 of the service card to be added is "Hebei Province". Then, in order to match the card information displayed by the weather card added to the picture 202 with the weather of the shooting location, as shown in Figure 5a(4), the user can click the control 404, and the mobile phone can respond to the user operation and refresh the display interface of the mobile phone from the display interface 401 shown in Figure 5(4) to the display interface 401 shown in Figure 5a(5).

[0239] For example, in response to the user clicking the control 404 shown in FIG5a(4), the mobile phone can obtain the metadata information of the picture 202 shown in FIG5a(1), and the metadata information may include the shooting time (here, 5:10 p.m. on June 5, 2020 as shown in FIG5a(1)), the shooting location and other information; then, the mobile phone can call the weather application to search for the historical weather information with the time of the shooting time and the location of the shooting location; then, the mobile phone can use the historical weather information to refresh the card information shown in the weather card 303 and the weather card 402 shown in FIG5a(4) to obtain the card information of the weather card 303 and the weather card 402 shown in FIG5a(5). Among them, the weather information displayed by the weather card 303 and the weather card 402 shown in FIG5a(5) is the weather information of Hebei Province on June 5, 2020.

[0240] In this embodiment, since the weather card 403 shown in FIG5a (4) displays the weather forecast information for Beijing for today, tomorrow, and the day after tomorrow, i.e., it is used to represent the weather forecast information for multiple days, the weather card 403 is not suitable for displaying the weather information of the picture 202.

[0241] Therefore, the weather card 403 is no longer included in the weather card shown in FIG. 5 a ( 5 ).

[0242] In addition, compared with Figure 5a(4), as shown in Figure 5a(5), the text "Real-time Weather" displayed on the display interface 401 is also refreshed to "Photo Weather" to indicate that the card information in the weather card displayed in Figure 5a(5) is all weather information about photos, here is weather information related to picture 202.

[0243] In addition, compared with FIG5a(4), as shown in FIG5a(5), the display text of the control 404 is switched from "matching photos" to "real-time weather". Then, when the user clicks the control 404 as shown in FIG5a(5), the mobile phone can restore the display interface to the display interface as shown in FIG5a(4) to display the real-time weather information of the mobile phone's location in the three previewed weather cards.

[0244] Next, as shown in FIG5a(5), the user clicks on the control 406, and the mobile phone can respond to the user operation and switch the display interface of the mobile phone to the display interface 201 shown in FIG5a(6).

[0245] Specifically, the mobile phone can respond to the user clicking the control 406 to execute the following S101a, S101b, S101c, S102, S103, and S104 as shown in Figure 3a, thereby converting the weather card 303 shown in Figure 5a (5) into the image 2051 shown in Figure 5a (6), and inserting the image 2051 between the foreground image 202a and the background image 202b of the image 202 to obtain the image 206.

[0246] The following describes S101a, S101b, S101c, S102, S103, and S104 shown in FIG3a with reference to FIG5a and FIG6a to FIG6c.

[0247] As shown in FIG. 3 a , after S100 a , the method further includes S101 a .

[0248] S101a: The mobile phone may divide the first image into a foreground image including foreground elements and a background image including background elements.

[0249] In conjunction with FIG. 5 a , please refer to FIG. 6 a , which shows a process of performing the segmentation operation S101 a on the picture 202 in FIG. 5 a to obtain a foreground image including foreground elements and a background image including background elements.

[0250] In S101a, the mobile phone may perform semantic segmentation and depth analysis on the image 202 shown in FIG6a(1) (here, an example of the first image) to obtain multiple layers, each of which includes different elements. For example, the multiple layers may include a layer including human elements, a layer including sky elements (here, blue sky and white clouds), and a layer including mountain elements. The areas of each layer except for the included elements are transparent areas, and the size of each layer is the same as the size of the image 202.

[0251] In S101a, the mobile phone may determine a background layer from the multiple layers based on the elements and / or semantics of each of the multiple layers, thereby obtaining a background image including background elements. For example, when determining a background layer based on the elements within the layers, the mobile phone may determine a layer with relatively simple elements and relatively few elements, such as a relatively pure color block, as the background layer.

[0252] For example, in the scene shown in FIG6a, the mobile phone can combine the semantics of the layers (such as the sky, the ground, the wall, the water surface, etc.) to determine that the layer including the sky element and the layer including the mountain element are both background layers; then, the mobile phone can synthesize the two background layers to obtain a background image 202b including background elements as shown in FIG6a (3), wherein the background image 202b includes background elements (blue sky, white clouds and mountain elements) 2022, and the area of ​​the background image 202b other than the background element 2022 (such as the area where the foreground element 2021 is located) is a transparent area.

[0253] In S101a, the mobile phone may determine the layers except the background layer among the obtained multiple layers as foreground layers, or determine the foreground layers based on the semantics of the layers to obtain a foreground image including foreground elements.

[0254] For example, in the scene shown in Figure 6a, the mobile phone can combine the semantics of the layers to determine the layer including the human element as the foreground layer, and determine the foreground layer as the foreground image 202a including the foreground element 2021. The foreground image 202a includes the foreground element (human element) 2021, and the areas in the foreground image 202a other than the foreground element 2021 (for example, the area where the background element 2022 is located) are all transparent areas.

[0255] In this way, the mobile phone can call the UI framework module through the gallery application to divide the picture 202 into a foreground image 202a including foreground elements and a background image 202b including background elements.

[0256] It should be understood that, in the application scenario shown in FIG5a, the timing for the mobile phone to execute S101a shown in FIG3a to segment the first image is not limited to responding to the user clicking the control 406 shown in FIG5a(5), but can also execute S101a in response to the user clicking the control 2041 shown in FIG5a(2). In other words, as long as the user selects the first image to be added, the mobile phone can segment the first image to obtain the corresponding foreground image and background image.

[0257] It should be understood that although in the application scenario related to Example 1 above, the first image (such as the picture 202 shown in Figure 5a) is only divided into a foreground image and a background image, in some embodiments, the image objects corresponding to the foreground element may be multiple, so the foreground images obtained by segmentation may also be multiple, for example, each foreground image includes one foreground element, and the other areas are transparent. Similarly, the first image may also be divided into multiple background images, for example, each background image includes one background element, and the other areas are transparent. The background elements involved in different background images may be different. For example, in the application scenarios of Figures 5a and 6b, the mobile phone may divide the image including the sky element into a background image, and the other areas are transparent; and the mobile phone may divide the image including the mountain element into a background image, and the other areas are transparent; if there are multiple people in the photo, at least one person can constitute a foreground image, and the area in the foreground image other than the at least one person is a transparent area.

[0258] It should be understood that the above embodiment uses a person as the foreground element. In other embodiments, the foreground element can be any type of image object, such as an animal, a plant, an object, or a building. This application does not limit the type of image object involved in the foreground element. Furthermore, the foreground element in the first image is not limited to the single person in the example above. The foreground element can be multiple image objects, such as multiple people, or people and animals, etc., and this is also not limited here.

[0259] In addition, the foreground image and the background image obtained by the mobile phone after segmenting the first image have the same format as that of the first image.

[0260] After S100b, the mobile phone may further execute S101b.

[0261] S101b: The mobile phone may convert the first service card into a second image.

[0262] Among them, the first service card is a control, not a picture. In order to embed the first service card into the first image, the mobile phone's gallery application can call the UI framework module to convert the first service card into a picture format to obtain a second image. The second image has the same format as the above-mentioned first image. In this way, images with the same format are easy to synthesize through the following S102.

[0263] For example, if the first image is in Scalable Vector Graphics (SVG) format, the mobile phone may convert the first service card into SVG format; and if the first image is in JPG format, the mobile phone may convert the first service card into JPG format.

[0264] The method of converting a service card into an image may differ based on the format of the first image. For example, if the image format of the first image is a bitmap image, the mobile phone can rasterize the first service card to convert the first service card represented in a vector graphics format into a bitmap.

[0265] Referring to FIG6b , the mobile phone can convert the weather card 303 shown in FIG5a (5) into a picture through the above-mentioned S101b to obtain image 205. After the weather card 303 is converted into image 205, the card information displayed in image 205 is the same as the card information displayed in the weather card 303. The difference is that image 205 is no longer a service card and cannot be interacted with.

[0266] Optionally, after S101b, the process may further include S101c.

[0267] S101c, the mobile phone may adjust the second image to a third image.

[0268] In some embodiments, the mobile phone can determine the first area of ​​the image in which the service card is inserted in the first image; and the mobile phone can refer to the image parameters of the first image related to the first area to adjust the corresponding image parameters of the service card in picture format (the second image here), so that after the second image with adjusted image parameters (named as the third image here) is inserted into the first image, the third image can be more integrated with the picture of the first image, so that the service card inserted into the first image can be displayed without being abrupt on the entire image.

[0269] In some embodiments, as described above, the first image of the service card to be added can be divided into a foreground image and a background image. Then, when the mobile phone determines the first area of ​​the image to which the service card is to be inserted, the service card in the corresponding picture format can be inserted between the foreground image and the background image, and the foreground element in the foreground image can block the area ratio of the service card within a preset threshold range, for example, 3% to 20% of the area of ​​the service card is blocked by the foreground element, so as to determine the position of the first area. The preset threshold range can be flexibly changed according to actual needs and is not limited here.

[0270] In some embodiments, the mobile phone may also determine the first area based on a location or area selected by the user in the first image. For example, if the user selects the coordinates of a service card to be added in the first image, the mobile phone may determine the first area including the coordinates based on the coordinates.

[0271] In some embodiments, the image parameters of the first image related to the first area may include but are not limited to: the size of the first area, the shape of the first area, the perspective parameters of the background elements and / or foreground elements located in the first area (which may include at least one of the perspective direction and the perspective angle), the color, hue, transparency, lighting characteristics and other information of the foreground elements and / or background elements related to the first area.

[0272] For example, the solution of adjusting the second image by the mobile phone with reference to the perspective parameters (which may include at least one of the perspective direction and perspective angle) of the background elements located in the first area will be described in detail in the application scenario 2 of Figures 7a to 7d below and will not be repeated here.

[0273] The method for adjusting the second image by reference to the perspective parameters of the foreground element located in the first area is similar to the principle of adjusting the second image by reference to the perspective parameters of the background element described above. For example, the location of the service card to be added in the first image (e.g., the first area described above) is not limited to being located between the background image where the background element is located and the foreground image where the foreground element is located. It can also be located in the layer above the foreground image where the foreground element is located (e.g., the top layer). In this way, the second image can be adjusted by reference to the perspective parameters of the foreground element and / or background element located in the first area.

[0274] The lighting characteristics may be at least one lighting characteristic such as direct lighting, reflected lighting, and shadow.

[0275] For example, the background image of the first image includes a sun element, and the light emitted by the sun element shines on the human element in the foreground image, and the insertion position of the second image converted from the first service card in the first image (such as the first area mentioned above) is on the lighting path. Then the mobile phone can use Ray Tracing (RT) technology to calculate the direct lighting characteristics of the area or position on the lighting path in the second image, and render the direct lighting characteristics of the position on the lighting path in the second image to obtain a third image, so that after the third image is inserted into the first area in the first image, the position on the lighting path in the third image can display a visual effect of direct lighting. Optionally, since calculating the lighting characteristics requires a high computing power, the mobile phone can interact with the server to implement the calculation of the direct lighting characteristics of the image, so as to obtain the rendering result of the third image, and thus obtain the third image using the rendering result.

[0276] Of course, the way in which a mobile phone renders a third image with a visual effect of corresponding lighting characteristics on a service card in picture format is not limited to the above example. Other technologies such as AIGC can also be used to calculate the lighting characteristics of the service card (or its converted picture).

[0277] Referring to FIG6c , FIG6c(1) and FIG6c(3) respectively illustrate foreground image 202a and background image 202b obtained through the segmentation process shown in FIG6a . Furthermore, as shown in FIG6c(1) and FIG6c(3), a first area to be inserted into foreground image 202a and background image 202b is indicated by a selection box, respectively, as shown in FIG6c(1) and FIG6c(3). The size of this first area is the same as the size of the image-format service card shown in FIG6b , i.e., image 205.

[0278] In this application scenario 1, as shown in FIG6c(3), the background elements in the first area are the sky and the mountain elements, and there is no tilt angle. Therefore, in this application scenario 1, the mobile phone does not need to adjust the perspective parameters of the image 205 shown in FIG6b.

[0279] In this application scenario 1, referring to FIG6c(1) and FIG6c(3), as well as FIG6b, the mobile phone can refer to the hue (optionally including color, transparency, etc.) of the background elements located in and around the first area, and the hue of the foreground elements located in and around the first area, to adjust the hue of the image 205 through S103c, so as to obtain the image 2051 shown in FIG6b and FIG6c(2). The hue of the image 2051 can match the hue of the foreground elements and background elements in and around the first area.

[0280] S102, the mobile phone synthesizes the foreground image including the foreground elements, the background image including the background elements and the third image (when S101c is not included, it is synthesized with the second image) into a target image, so that the above-mentioned first image can be refreshed as the target image.

[0281] Among them, the synthesized target image may include three layers from top to bottom, namely, an upper layer (also referred to as the first layer below), a middle layer (also referred to as the second layer below), and a lower layer (also referred to as the third layer below).

[0282] Among them, the foreground image is the upper layer, the third image (or the second image) is the middle layer, and the background image is the lower layer.

[0283] In application scenario 1, please continue to refer to Figure 6c. When the mobile phone synthesizes the third image after tone adjustment into the first image, the foreground image 202a obtained by segmentation through the process shown in Figure 6a can be used as the first layer shown in Figure 6c(1), the background image 202b can be used as the third layer shown in Figure 6c(3), and the tone-adjusted image 2051 can be used as the second layer shown in Figure 6c(2). In order from top to bottom, the three images are synthesized through S102 to obtain image 206 (an example of the target image) shown in Figure 6c(4).

[0284] As shown in FIG6c(4), the lower right corner of the image 2051 inserted in the image 206 is blocked by the foreground element (here, the head of the person) in the foreground image 202a, and the image 2051 can block the background element located in the first area of ​​the background image 202b.

[0285] In other embodiments, the mobile phone may not split the first image, and may directly insert a service card in picture format, such as the second image or the third image, into the corresponding position of the first image. Then, the image of the service card may be located on the upper layer of the first image.

[0286] Because the mobile phone refers to the image parameters (here is the color tone) of at least one element among the foreground elements and background elements related to the first area of ​​the first image, it adjusts the color tone of the service card (here is the second image) converted into the picture format shown in Figure 6b, so as to embed the color-adjusted image 2051 into the first image. In this way, the picture of the service card embedded in the first image can be integrated and matched with the color tones of the surrounding image elements.

[0287] Returning to FIG. 5 a , the mobile phone can display the synthesized image 206 shown in FIG. 6 c ( 4 ) on the display interface 201 shown in FIG. 5 a ( 6 ) in response to the user clicking the control 406 shown in FIG. 5 a ( 5 ).

[0288] In some embodiments, the image 206 in the display interface 201 shown in FIG5a(6) may be a replacement result of the image 202 shown in FIG5a(1). In some embodiments, the image 206 may also be a picture saved in a gallery application.

[0289] Optionally, after S102 , the process may further include S103 .

[0290] S103 , the mobile phone may determine card metadata related to the target image based on the target image, the third image, and the first service card.

[0291] The card metadata may include but is not limited to at least one of the following: card type, card information, card size (such as the size of the first area mentioned above), card position (such as the position of the first area in the first image), card shape (such as the shape of the first area mentioned above), and card image parameters.

[0292] Optionally, the card metadata may also include segmentation data related to S101a above, which indicates how to segment the first image to obtain a foreground image and a background image. Thus, in the process of FIG3b , when the mobile phone instantiates a second service card for the target image, the mobile phone can refer to the segmentation data to add the instantiated second service card between the foreground image and the background image in the target image. Furthermore, in the process of sharing the target image shown in FIG3c , mobile phone 2 does not need to configure a segmentation algorithm to segment the target image. Instead, it only needs to use the segmentation data in the card metadata to obtain the foreground image and the background image to instantiate the service card.

[0293] The image parameters of the card are the image parameters of the third image after the service card in picture format is adjusted to the third image through the above S101c, or the image parameters of the card can also be the image adjustment parameters (such as size adjustment parameters, transparency adjustment parameters, etc.) used when the service card in picture format is adjusted to the third image through the above S101c, wherein the image adjustment parameters can indicate information such as the adjustment direction and adjustment amount of the image parameters. The image adjustment parameters are relative to the above image parameters, and the image adjustment parameters are absolute image parameters.

[0294] In some embodiments, the image parameters may include, but are not limited to, information such as color, hue, transparency, lighting characteristics, and perspective parameters.

[0295] For example, the mobile phone can determine the card type and card information of the first service card based on the first service card determined in S100b, and the mobile phone can determine the card perspective parameters, card size, card position, card color, card tone, card lighting characteristics, card transparency, etc. in the card metadata based on the perspective parameters, size of the third image and the synthesis position (the above-mentioned first area), color, hue, lighting characteristics, transparency, etc. of the third image used when performing the synthesis operation of S102.

[0296] For example, in application scenario 1 shown in FIG5a and FIG6a to FIG6c, the card metadata related to image 206 determined by the mobile phone includes: the card type is a weather card, the card information is the weather information displayed by image 2051 shown in FIG6c (4), the card size is the size information of the image 2051, the card position is the display position of image 2051 (here, the position between the foreground image 202a and the background image 202b in the first area indicated by the dotted box shown in FIG6c), and the image parameters of the card (here, the color tone parameters, such as the color tone parameters of image 2051 after the color tone of image 205 is adjusted in FIG6b).

[0297] Optionally, after S103 , the method may further include S104 .

[0298] S104: The mobile phone may write the card metadata into the metadata of the target image.

[0299] Among them, the mobile phone can write the card metadata obtained in the above S103 into the metadata of the image 206 shown in Figure 5a (6), so that the metadata of the image 206 embedded with the service card in image form can carry the metadata information of the service card indicated by the target image in which it is embedded.

[0300] Alternatively, in other embodiments, the mobile phone may also write the obtained card metadata into a file with the same name as the image 206 , so that the card metadata can be stored in association with the image 206 .

[0301] It should be understood that the embodiments of Figures 5a, 6a to 6c are illustrated by taking the service cards added to the pictures as system preset service cards as an example, but the service cards involved in the various embodiments of the present application are not limited to system preset service cards, but can also be service cards generated by other known or future developed methods, which are not limited here.

[0302] For example, a mobile phone may generate a service card related to the content of a movable control in response to a user's operation on the movable control. This service card can be used as a service card for adding to an image in various embodiments of the present application. The control content may include information displayed on the movable control and / or information related to the movable control but not displayed. The movable control may be a notification message, a floating control, a transfer station control, etc.

[0303] In an embodiment of the present application, in order to add a service card to a first image, the mobile phone may first convert the service card to be added into a picture format, and then add the service card in picture format (such as the second image or the third image shown in FIG3a ) to the first image to obtain a target image, and associate the metadata information of the service card with the target image and save it. In this way, each time the target image needs to be displayed, the mobile phone may refer to the metadata information, create a service card with the same display effect as the second image (or the third image), and add the service card to the target image, so that the interactive service card is displayed at the position where the service card in picture format is displayed in the target image, and the instantiation of the service card in the image (such as creating a service card) can be quickly realized, so that the instantiated service card in the target image can be interactive.

[0304] In conjunction with Figures 3a, 5a, and 6a through 6c above, after the process in Figure 3a, please refer to Figure 3b, which illustrates the process of instantiating a service card for the target image generated in Figure 3a by the mobile phone. In this embodiment, application scenario 1 of the process in Figure 3b will be described in conjunction with Figures 5b and 6d.

[0305] Referring to FIG. 3b , the process may include the following steps:

[0306] S300: The mobile phone determines the target image.

[0307] For example, after the mobile phone displays the display interface 201 shown in Figure 5a (6), the mobile phone switches the application running in the foreground from the gallery application to other applications. After a while, the user opens the gallery application to display the display interface 407 shown in Figure 5b (1).

[0308] Display interface 407 is an application interface of the gallery application, and includes reduced images of multiple images stored in the gallery application, including image 2061. Image 2061 is a reduced image of image 206 stored in the gallery application through the process of Figures 5a and 6a to 6c.

[0309] The image 206 pointed to by the dotted arrow in FIG5b(1) is the original image after the image 2061 is enlarged. The image 206 is the same as the image 206 shown in FIG5a(6).

[0310] Then, the user clicks on the image 2061 shown in FIG5b(1) to request the display of the image 206 (an example of the target image). The mobile phone can determine the image 206 to be displayed in response to the user's click operation, and execute the following S301 to S303, thereby switching the display interface of the mobile phone from the display interface 407 shown in FIG5b(1) to the display interface 201 shown in FIG5b(2) to display the image 206, wherein the position where the image 2051 is displayed in the displayed image 206 (for example, the first area mentioned above) is instantiated with a service card, which is the weather card 309 in this case.

[0311] The following describes the process of switching the display interface of the mobile phone from the display interface 407 shown in FIG5b(1) to the display interface 201 shown in FIG5b(2) in combination with S301 and S303 as well as FIG6d.

[0312] S301: The mobile phone obtains card metadata related to the target image.

[0313] As described in S104 shown in FIG3a , the card metadata is stored in the image metadata of the target image. Therefore, the mobile phone can extract the card metadata from the image metadata of the original image (i.e., image 206 ) corresponding to image 2061 shown in FIG5b ( 1 ).

[0314] In some embodiments, when the card metadata related to the image 206 is not saved or the saved card metadata is incomplete in the process shown in FIG. 3 a , the mobile phone may further obtain the card metadata by performing image analysis on the image 206 .

[0315] As described above, the card metadata may include, but is not limited to, at least one of the following: card type, card information, card size (e.g., the size of the first region), card position (e.g., the position of the first region in the first image), card shape (e.g., the shape of the first region), and card image parameters. In some embodiments, the image parameters may include, but are not limited to, information such as color, hue, transparency, lighting characteristics, and perspective parameters.

[0316] For example, the mobile phone can analyze the image 206 shown in FIG5b(1) to determine that the rectangular frame image 2051 is a service card image; and the mobile phone can perform semantic analysis on the image elements in the image 2051, such as the sun element, the temperature element, and the place name element, to determine that the image 2051 is a weather card image, thereby determining the card type as a weather card. The mobile phone can also use the same or similar methods to determine the card information, card size, card position, card shape, card image parameters, etc.

[0317] Compared with obtaining card metadata through image sharing, reading the card metadata related to the target image is more accurate and efficient. You can flexibly choose the corresponding method to obtain card metadata based on the computing power of the mobile phone and actual needs.

[0318] S302: The mobile phone may instantiate a second service card for the target image based on the card metadata, so that the interactive second service card is displayed on the target image.

[0319] For example, as shown in Figure 6d (1), the mobile phone can create a second service card (for example, calling the interface of the weather application for creating a service card) based on the card metadata related to image 2051, which is weather card 309 here, wherein the size, shape, card information, image parameters, etc. of weather card 309 are the same as the corresponding parameters of image 2051.

[0320] The image parameters may include but are not limited to: color, hue, transparency, lighting characteristics, perspective parameters and other information.

[0321] In one example, a mobile phone may call an interface of a weather application for creating a service card, and use the card size and card information as parameters to create a weather card with the card information and a card size that is the card size in the parameters; then, the mobile phone adjusts the shape and image parameters of the created weather card based on the card shape and image parameters in the card metadata to obtain the weather card 309.

[0322] In another example, the mobile phone calls an interface for creating a weather service card. For example, if the interface supports inputting card shape and image parameters, the interface can carry the above card metadata, so that the above weather card 309 can be directly created.

[0323] In this embodiment, the display effect of the image 2051 is the same as the display effect of the weather card 309 instantiated by the mobile phone, but the weather card 309 is an interactive service card, and the image 2051 is just a picture format of the weather card 309.

[0324] In other embodiments, the display effect of the weather card 309 may be different from that of the image 2051. For example, the line color and thickness of the rounded rectangular outline of the weather card 309 may be different from those of the rounded rectangular outline of the image 2051.

[0325] In this way, when the mobile phone displays the image 206 shown in Figure 5a (6), the mobile phone executes the process of Figure 3b on the image 206 to instantiate the weather card 309 at the position where the image 2051 is located in the image 206, so as to switch the display interface from the display interface 201 shown in Figure 5a (6) to the display interface 201 shown in Figure 5b (2). In this way, the user can see the process of the image 2051 changing into the weather card 309 from the image 206 displayed in the display interface 201 of the mobile phone. For example, there is a difference in the color of the lines of the rounded rectangular outline between the image 2051 and the weather card 309. In addition, in some embodiments, when the mobile phone switches the display interface of the mobile phone from the display interface 407 shown in Figure 5b (1) to the display interface 201 shown in Figure 5b (2) in response to the user operation of the user clicking on the image 2061 shown in Figure 5b (1), the mobile phone may first switch the display interface from the display interface 407 shown in Figure 5b (1) to the display interface 201 shown in Figure 5a (6) to display the original image corresponding to the image 2061, that is, the image 206, and then switch the display interface from the display interface 201 shown in Figure 5a (6) to the display interface 201 shown in Figure 5b (2) to instantiate a weather card 309 at the corresponding card position of the image 206.

[0326] Whether the display effects of the image 2051 and the weather card 309 are the same can be flexibly configured according to actual needs, and this application does not impose any restrictions.

[0327] As shown in FIG6d , the mobile phone can insert the weather card 309 created as shown in FIG6d (1) into the image 206 (an example of the target image) to obtain the image 206 (here an example of the target image) with the interactive weather card 309 as shown in FIG6d (2).

[0328] Specifically, as shown in FIG6d(1), the mobile phone can insert the weather card 309 between the first layer shown in FIG6c(1) and the second layer shown in FIG6c(2) according to the card position in the card metadata, that is, the weather card 309 is inserted between the foreground image 202a and the image 2051 of the weather card in picture format, and the insertion position of the weather card 309 is the first area indicated by the dotted box shown in FIG6c. In this way, a service card, specifically the weather card 309, can be instantiated at the position where the image 2051 was originally displayed in the image 206 shown in FIG6d(2), so that the weather card 309 can cover the image 2051. In this way, when the user operates the weather card 309 shown in FIG5b(2) (i.e., the weather card 309 shown in FIG6d(2)), the mobile phone can perform interactive operations related to the service card 309 to display the interactive results related to the service card 309. For example, if the user clicks on the weather card 309 shown in FIG5b(2), the mobile phone may respond to the user operation by calling the weather application through the weather card 309 to display the application interface of the weather application, which may be an example of the above-mentioned interaction result.

[0329] For example, the application interface may include weather information related to the weather information displayed in weather card 309, such as displaying the weather forecast information for Hebei Province for the past three days. Alternatively, the application interface may display more detailed information about the weather information in weather card 309, such as including not only the location information and temperature information displayed in weather card 309, but also information related to the location, such as the wave index and clothing recommendations. Alternatively, the application interface may display forecasted weather information related to the current location of the mobile phone. This is not a limitation.

[0330] In the application scenario of FIG. 6 d , the mobile phone can create a new second service card based on the card metadata, and insert the second service card into the corresponding position of the target image to instantiate the service card for the target image.

[0331] In other embodiments, the mobile phone may further add a touch event to a corresponding object in a service card in picture format displayed in the target image (eg, image 2051 shown in FIG. 6 d ) to instantiate a second service card in the target image.

[0332] For example, the mobile phone can add a touch event to the location information (e.g., "Hebei") in the image 2051 shown in FIG5a(6). The touch event can call the interface of the weather application that displays weather information (the location information can be passed into the interface as a parameter), thereby realizing the process of instantiating the service card 309 for the image 2051, so that the display interface of the mobile phone can be switched from FIG5a(6) to FIG5b(2). When the user clicks the "Hebei" object in the weather card 309 shown in FIG5b(2), the touch event can be triggered to display the information interface about the weather forecast for Hebei provided by the weather application.

[0333] It should be understood that the process shown in FIG3b is triggered when the target image needs to be displayed. In other words, the instantiation process of the service card shown in FIG3 is single-shot. That is, each time a user requests the display of a target image through an operation, the mobile phone needs to execute the above process shown in FIG3b once to instantiate a service card for image 206 based on the card metadata. In this way, a service card can be displayed between the foreground and background images of image 206, and the card information, size, shape, perspective parameters, etc. of the service card are the same as the information of the service card in the image format originally displayed in the target image. When the target image with the second service card instantiated is canceled (for example, the user switches the image displayed in the gallery application from the target image to another image), the mobile phone can cancel the second service card instantiated on the target image and close the process of the mobile phone executing the process of FIG3b, to ensure that the second service card displayed on the target image is generated in real time each time the target image is displayed.

[0334] In some embodiments, FIG5c shows a schematic diagram of a sharing scenario of an image with a service card (or a service card with a picture format).

[0335] Please refer to Figure 5c. The display interface 201 of Figure 5c(1) is the image display interface of the gallery application. The display interface 201 is the same interface as the display interface 201 shown in Figure 5b(2) above. The user clicks the control 207 shown in Figure 5c(1). As shown in Figure 5c(2), the mobile phone can respond to the user operation and display a window 408 on the display interface 201. The window 408 may include one or more controls, which may include but are not limited to: image 206 and thumbnails of other pictures stored in the gallery application (such as image 4091, image 4092), information application icon 410, etc.

[0336] The image 206 shown in FIG5c(2) is in a selected state as indicated by an arrow, indicating that the image 206 is the image to be shared. The enlarged view of the image 206 shown in FIG5c(2) is the image 206 shown in FIG5a(6), that is, the image 206 to be shared does not include the instantiated service card 309 shown in FIG5c(1). The image 206 to be shared is still the image 206 with the service card (i.e., image 2051) in the image format stored in the gallery application.

[0337] In other embodiments, when the user clicks the control 207 shown in FIG. 5a (6), the mobile phone will also respond to the user operation and display the window 408 shown in FIG. 5c (2) to share the image 206 shown in FIG. 5a (6).

[0338] Continuing with reference to FIG5c(2), the user clicks on the message application icon 410. In response to the user operation, the mobile phone switches the display interface from the display interface 201 shown in FIG5c(2) to the display interface 412 shown in FIG5c(3). The display interface 412 displays a list of contacts that the user has recently chatted with using the message application, which are contact 1 indicated by control 413 and contact 2 indicated by control 414.

[0339] Then, the user clicks control 413, as shown in FIG5c(3). In response to the user operation, the mobile phone displays a window 415 on the display interface 412. The window 415 may include, but is not limited to, a thumbnail 4151, a send control 4152, and other controls. Thumbnail 4151 is a thumbnail of image 206 shown in FIG5a(6).

[0340] Next, the user clicks on control 4152, and the mobile phone can respond to the user operation by switching the display interface from display interface 412 shown in Figure 5c (3) to display interface 416 shown in Figure 5c (4). Display interface 416 is a conversation interface for chatting between the user of the mobile phone (also called mobile phone 1) and contact 1, wherein the display interface 416 displays the image 206 shared by the user of mobile phone 1 to contact 1 (for example, using mobile phone 2). In order to facilitate the user to view the shared image 206 in the display interface 416, Figure 5c (4) indicates the enlarged screen of the shared image 206 with a dotted arrow. Among them, the image 206 shared by mobile phone 1 to mobile phone 2 is the same as the image 206 shown in Figure 5a (6), that is, the image 206 shared by the user to contact 1 has a service card in picture format, that is, image 2051.

[0341] In addition, as shown in FIG3c , when mobile phone 1 responds to the user clicking on the control 4152 shown in FIG5c(3), mobile phone 1 can send the target image (here, image 206 shown in FIG5a(6)) and the metadata of the target image to mobile phone 2 used by contact 1. Since the card metadata related to the image 206 is written into the metadata of the image 206, the card metadata related to the image 2051 is also shared with mobile phone 2.

[0342] After the process in Figure 5c, please refer to Figure 5d, which shows a schematic diagram of the interface of mobile phone 2 used by the above-mentioned contact 1. As shown in Figure 5d(1), mobile phone 2 displays the application interface of the information application, here the display interface 416, which includes the conversation content between the user of mobile phone 2 (i.e., the above-mentioned contact 1) and contact 3, wherein contact 3 is the user of mobile phone 1 shown in Figure 5c. As shown in Figure 5d(1), the display interface 416 includes an image 206 shared by the user of mobile phone 1 (contact 3) to the user of mobile phone 2 (contact 1), and the image 206 may include a foreground image 202a, a background image 202b, and an image 2051 inserted between the foreground image 202a and the background image 202b.

[0343] Then, the user clicks on the image 206 shown in Figure 5d(1), and the mobile phone 2 can respond to the user operation and execute the process shown in Figure 3b to use the card metadata in the metadata of the image 206 to instantiate the weather card 309 for the image 206, and switch the display interface of the mobile phone from the display interface 416 described in Figure 6d(1) to the display interface 417 described in Figure 6d(2), wherein the display interface 417 displays the image 206 instantiated with the weather card 309.

[0344] In this way, in this embodiment, since the card metadata generated in the process of Figure 3a is written into the metadata of the target image, when mobile phone 1 shares the target image with the service card to other electronic devices (such as mobile phone 2), the card metadata can be shared to other electronic devices through the metadata of the target image. In this way, mobile phone 2 can use the shared target image and its metadata to use the process of Figure 3b to instantiate the service card when the target image needs to be displayed, so that the target image can display an interactive service card.

[0345] In some embodiments, the card metadata is carried in the metadata of the target image, so that the data shared from mobile phone 1 to mobile phone 2 includes the target image and its metadata. In this way, the file data only has one file where the metadata of the target image is located, and fewer files are transmitted, which is conducive to data sharing.

[0346] In another embodiment, the card metadata of the target image can also be stored separately instead of being written into the metadata of the target image. In this way, when the target image is shared by a mobile phone, the shared file data includes not only the metadata of the target image, but also the card metadata associated with the target image. In this way, multiple file data are transmitted between different electronic devices.

[0347] In actual applications, the mobile phone can flexibly determine whether to store the card metadata separately as a file or write it into the metadata of the target image according to needs, and there is no limitation here.

[0348] Application Scenario 2

[0349] The above application scenario 1 takes Figures 5a to 5d and Figures 6a to 6d as examples to describe the implementation process of Figures 3a, 3b, and 3c of the present application.

[0350] In this application scenario 2, FIG. 7 a to FIG. 7 d are taken as an example to describe the implementation process of FIG. 3 a to FIG. 3 b above.

[0351] The implementation process of this application scenario 2 is mostly the same as that of the above application scenario 1, with the following three differences:

[0352] Difference 1: When determining the first service card through S100b shown in FIG. 3a , the mobile phone may recommend service cards matching the first image, so that the user can select the first service card to be added to the first image from the recommended service cards.

[0353] Difference 2: When the mobile phone adjusts the second image through S101c shown in Figure 3a, it can adjust the perspective parameters of the second image so that the perspective parameters of the adjusted third image match the perspective parameters of the background elements in the background image obtained by segmenting the first image.

[0354] Difference 3: When the mobile phone instantiates the second service card for the target image through S302 shown in Figure 3b, the mobile phone can first create the above-mentioned first service card based on the card information and card type in the card metadata, and then adjust the first service card in combination with the perspective parameters, card size, card shape and other information in the card metadata to obtain a second service card with the same perspective parameters, size and shape as the service card in the image format displayed in the target image. Finally, the second service card is inserted into the corresponding card position in the target image.

[0355] The following describes the application scenario 2 shown in FIG. 7 a to FIG. 7 d in conjunction with FIG. 3 a to FIG. 3 b , focusing on the differences between the application scenario 2 and the above-mentioned application scenario 1, and omitting the similarities.

[0356] Please refer to FIG7a . As shown in FIG7a (1), the display interface 201 of the mobile phone is the application interface of the gallery application. The display interface 201 may include, but is not limited to, an image 502 and controls 203 and 207. The image 502 depicts a person walking in a gym, with the background being the curved windows of the building where the gym is located.

[0357] When the user clicks on the control 203, the mobile phone may respond to the user operation by displaying the display interface 201 shown in FIG7a(2). The display interface 201 may further include a control 204. When the user clicks on the add card option 2041 in the control 204, the mobile phone may respond to the user operation by determining the type of service card (here, a music card and a sports card) that matches the screen scene of the image 502 shown in FIG7a(2), and based on the type of the service card, switch the display interface to the display interface 3010 shown in FIG5a(3) to recommend adding a music card or a sports card to the user.

[0358] As shown in FIG7a(3), the display interface 3010 may include, but is not limited to, a search control 301, a control 306 for adding a music card of a music application, and a control 308 for adding a sports card of a sports health application. Control 306 is a control related to the music card, and control 308 is a control related to the sports card.

[0359] As shown in FIG7a(3), the user clicks control 308, and the mobile phone can respond to the user operation by switching the display interface to display interface 401 as shown in FIG7a(4). Display interface 401 may include two sports cards for preview, namely sports card 503 and sports card 504. Among the two weather cards previewed by the mobile phone, sports card 503, which is displayed on the top layer, represents the currently selected weather card.

[0360] In addition, as shown in FIG7a(4), the display interface 401 also displays the text “Daily Exercise” to indicate that the card information displayed on each exercise card in the exercise card 503 and the exercise card 504 may include: information on the number of exercise steps of the user on that day recorded by the sports health application.

[0361] As shown in FIG7a(4), the user clicks on the control 406, and the mobile phone can respond to the user operation to execute the above-mentioned S101a, S101b, S101c, S102, S103, and S104 as shown in FIG3a, thereby converting the sports card 503 shown in FIG7a(4) into the image 5051 shown in FIG7a(5), and inserting the image 5051 between the foreground image 502a and the background image 502b after the image 502 is segmented as shown in FIG7a(1), so as to obtain the image 506 shown in FIG7a(5).

[0362] Then, the mobile phone executes S302 shown in FIG3b on the image 506 shown in FIG7a(5) to instantiate a sports card 509 on the image 506 to switch and display the image 506 with the sports card 509 shown in FIG7a(6).

[0363] FIG7b shows a schematic diagram of the process of the mobile phone segmenting the image 502. As shown in FIG7b, the mobile phone can segment the image 502 shown in FIG7b(1), which is also the image 502 shown in FIG7a(1) (an example of the first image), through S101a shown in FIG3a to obtain the foreground image 502a with the foreground element 5021 and the background image 502b with the background element 5022 shown in FIG7b(2). Among them, the area of ​​the foreground image 502a other than the foreground element 5021 (the walking figure element) (for example, the area where the background element 5022 is located) is all transparent area. And the area of ​​the background image 502b other than the background element 5022 (the window of the building where the gym is located) (for example, the area where the foreground element 5021 is located) is all transparent area. For the convenience of readers' understanding, the transparent area in the background image 502b shown in FIG7b(2) is shown with a dotted line.

[0364] FIG7c shows a process in which the mobile phone processes the sports card 503 shown in FIG7a(4) through S101b and S101c to obtain the image 5051 (an example of the third image) shown in FIG7c.

[0365] First, as shown in FIG7c(1), the mobile phone can convert the sports card 503 shown in FIG7a(4) into a picture format through S101b to obtain an image 505 (an example of the second image).

[0366] Then, the mobile phone can determine the first area of ​​the service card to be displayed in the image 502 (an example of the first image) shown in Figure 7c(2), which is also the image 502 shown in Figure 7a(1) and Figure 7a(2), and the first area 5023 is shown in a dotted box in Figure 7c(2).

[0367] As shown in FIG7c(2), the first area 5023 displays an image of a curved window glass. Therefore, the background element corresponding to the first area (here, the window glass element) has perspective parameters, which may include a perspective direction and a perspective angle.

[0368] The perspective direction is the normal vector perpendicular to the plane, and the perspective angle is the wide angle of the photo or the shooting focal length.

[0369] Next, please refer to Figure 7c(1). The mobile phone can adjust the shape, size, and perspective parameters of the image 505 based on the perspective parameters of the background elements of the first area 5023 and the shape and size of the first area, so that the perspective parameters of the adjusted image 5051 (an example of the third image) are the same as the perspective parameters of the background elements displayed on the first area 5023 shown in Figures 7c(1) and 7c(2), and the shape and size of the image 5051 are also the same as the shape and size of the first area 5023.

[0370] Finally, the mobile phone can refer to the principle of the synthesis process shown in Figure 6c in application scenario 1 to synthesize the image 5051 shown in Figure 7c(1) into the first area (the first area shown in Figure 7c(2)) between the foreground image 502a and the background image 502b shown in Figure 7b(2) to obtain the image 506 shown in Figure 7a(5) (an example of the target image).

[0371] Furthermore, the mobile phone can write the perspective parameters, shape, size and related card type (here, sports card) of the adjusted image 5051 as card metadata into the metadata of the image 506 shown in FIG7a(5).

[0372] In this way, when adding a service card to an image, when the image elements (not limited to background elements, but also including foreground elements, for example, when there are multiple foreground images) displayed at the position where the service card is to be added to the image have a certain tilt angle, such as curved glass, the mobile phone can refer to the tilt angle (specifically the perspective direction and perspective angle) to adjust the tilt angle of the service card in the picture format to be added to the image, so that the service card in the picture format that is attached to the position in the image is more closely fitted to the plane where the background elements at the attachment position are located in the image, so that the service card in the picture format added to the target image can be more integrated with the relevant foreground elements and / or background elements in the target image.

[0373] FIG. 7 d exemplarily shows the process in which the mobile phone instantiates a sports card 509 for the first area in the image 506 shown in FIG. 7 a ( 5 ).

[0374] First, as shown in FIG7d, the mobile phone can call the card creation interface of the sports health application through S302 shown in FIG3b based on the card metadata about the image 5051 related to the image 506 shown in FIG7a(5) to create a sports card 503. The sports card 503 can be the same as the sports card 503 shown in FIG7a(4). The mobile phone can pass the card type and card information in the card metadata to the above interface to create the sports card 503.

[0375] Then, the mobile phone can adjust the size, shape, and perspective parameters of the sports card 503 based on the card size, card shape, and card perspective parameters in the above card metadata through S302 shown in Figure 3b to obtain a second service card, which is the sports card 509 here.

[0376] Finally, referring to FIG7a(5), the mobile phone can insert the sports card 509 into the first area between the foreground image 502a and the image 5051 of the image 506 (specifically, the first area 5023 shown in FIG7c(2)) to obtain the image 506 shown in FIG7a(6) with the sports card 509 instantiated in the first area.

[0377] In some embodiments, the mobile phone may also directly switch the display interface to the display interface 201 shown in FIG7a(6) in response to the mobile phone clicking the control 406 shown in FIG7a(4) without displaying the image 506 shown in FIG7a(5). However, the target image stored in the mobile phone is still the image 506 shown in FIG7a(5). The implementation principle of this process is similar and will not be repeated here.

[0378] In an embodiment of the present application, a mobile phone can recommend service cards that match the content of an image to be added, thereby making the type of service card added to the image more compatible with the content and scene of the image. Then, when adding a service card in picture format to the image, the mobile phone can refer to the perspective parameters of the image elements displayed in the first area of ​​the service card to be added, optionally including shape, size, transparency, lighting characteristics, hue, transparency, etc., to further adjust the service card in picture format, so that the image of the adjusted service card in picture format (i.e., the third image) can be more closely aligned with the image elements related to the first area of ​​the first image and have a stronger sense of integration. Similarly, when instantiating a service card for a target image with a service card in picture format, the parameters of the created service card of the corresponding type can be adjusted by referring to the perspective parameters, shape, size, transparency, lighting characteristics, hue, etc. of the third image, and then the adjusted service card is attached to the first area of ​​the target image, so that the service card instantiated on the target image can be more closely aligned with the image elements related to the first area of ​​the target image and have a stronger sense of integration.

[0379] Application Scenario 3

[0380] In the above-mentioned application scenarios 1 and 2, the implementation process of the method of the present application is mainly illustrated by taking the example that a mobile phone can add service cards to a first image stored locally, and the added service cards are service cards selected by the user and / or recommended by the system based on the first image.

[0381] In this application scenario 3, FIG. 8a and FIG. 8b are taken as examples to describe the implementation process of FIG. 3a to FIG. 3b .

[0382] The implementation process of this application scenario 3 is similar to that of the above application scenarios 1 and 2. The differences are as follows:

[0383] Difference 1: When the mobile phone determines the first image of the service card to be added through S100a shown in FIG. 3a , the image currently captured by the mobile phone can be used as the first image.

[0384] Difference 2: When the mobile phone determines the first service card in S100b shown in FIG3a , the mobile phone may determine the first service card based on the real-time context. For example, when the mobile phone captures the first image or selects the first image to add a service card, if the real-time context is that the mobile phone is playing music, the mobile phone may recommend a music card based on the real-time context (playing music).

[0385] Difference 3: After the mobile phone obtains the target image with the instantiated service card through the process of Figures 3a and 3b, it can respond to user operations on the service card displayed on the target image by refreshing the card information of the service card and / or calling a music application to play music. Of course, the instantiated service cards displayed on the target image in Application Scenarios 1 and 2 above are also interactive, and the principles are the same.

[0386] 3a and 3b , the application scenario 3 shown in FIG. 8a and FIG. 8b will be described below, with emphasis on the differences between the application scenario 3 and the above-mentioned application scenario 1 and application scenario 2, and the similarities will not be repeated.

[0387] Please refer to Figure 8a. As shown in Figure 8a(1), the display interface 600 of the mobile phone is the application interface of the camera application. The display interface 600 may include but is not limited to: a shooting preview interface 608, a shooting button 602, a control 603 for displaying the currently captured image, and a control 604 for switching the shooting camera (for example, switching between the rear camera and the front camera). Among them, the current shooting preview interface 608 displays an image screen 601 captured by the rear camera. In addition, the applications running in the background of the mobile phone include a music application, which is playing an audio file named "Song 1".

[0388] As shown in FIG8a(1), when a user clicks the capture button 602, the mobile phone can capture the image screen 601 displayed on the capture preview interface 601 in response to the user operation to obtain a first image that is identical to the image screen 601. Since the first image is identical to the image screen 601, for ease of explanation, this embodiment uses image 601 to represent the first image.

[0389] And the mobile phone can determine that the service card to be added to the image 601 is a music card in response to the user clicking the shooting button 602, and create the music card by executing the above S101a, S101b, S101c, S102, S103, and S104 shown in Figure 3b, and convert the music card into the image 607 shown in Figure 8a (3), and insert the image 607 into the image 601 to obtain the image 605 shown in Figure 8a (3), wherein the image 605 is the original image of the music card captured and inserted in the picture format, and the mobile phone can store the image 605 in the gallery application. The mobile phone can switch the display interface from the display interface 600 shown in Figure 8a(1) to the display interface 600 shown in Figure 8a(2). The shooting preview interface 608 in the display interface 600 displays the image screen 601' currently captured by the camera. In addition, the control 603 shown in Figure 8a(2) can display a thumbnail 6031 of the image 605 obtained by this shooting as shown in Figure 8a(3).

[0390] In order to make it easier for readers to see the thumbnail 6031 clearly, Figure 8a(4) shows an enlarged effect of the thumbnail 6031. Compared with the original image corresponding to the thumbnail 6031 shown in Figure 8a(3), that is, image 605, the thumbnail 6031 is an image that is cropped and reduced according to the circular view.

[0391] The following describes the process of the mobile phone generating the image 605 shown in Figure 8a (3). In response to the user operation of clicking the shooting button 602 shown in Figure 8a (1), the mobile phone can determine that the mobile phone is running a music application to play an audio named "Song 1" (a real-time scenario); then, the mobile phone can determine that the first service card to be added to the captured image 601 is a music card, and obtain the name of the audio being played, here "Song 1", and information such as the playing progress of the audio through the music application. Then, the mobile phone can call the interface for creating a music card and pass the audio name, audio playing progress and other information as parameters to the interface to create a music card. The card information displayed on the music card is the same as the information shown in the image 607 shown in Figure 8a (3). Then, the mobile phone can convert the music card into a picture format to obtain the music card in the picture format shown in Figure 8a (3), that is, image 607. Then, the mobile phone inserts the image 607 between the foreground image and the background image of the above image 601 to obtain the image 605 shown in Figure 8a (3).

[0392] Among them, the foreground image is an image generated after the mobile phone performs a segmentation operation on the image 601 through S101a, and includes the foreground element 6051 shown in Figure 8a (here is the sky element), and the area other than the foreground element 6051 is a transparent area; the background image is an image generated after the mobile phone performs a segmentation operation on the image 601 through S101a, and includes the background element 6052 shown in Figure 8a (here is an image element composed of multiple trees), and the area other than the background element 6052 is a transparent area.

[0393] Thus, as shown in FIG8a(3), an image 605 (an example of a target image) captured and processed by the mobile phone displays an image 607 (a music card in picture format), and the image 607 displays the above-mentioned audio name, here "Song 1", as well as a poster image of the audio, and buttons for controlling the audio. However, the image 607 inserted into the image 605 shown in FIG8a(3) and FIG8(4) cannot perform card interactions, such as audio control. The mobile phone can also store the card metadata related to the image 607 in the metadata of the image 605. The card metadata may include: the card type is a music card, the card information includes the audio name "Song 1", the song progress is the above-mentioned "play progress", the poster image, and other information.

[0394] Then, as shown in FIG8a(2), the user clicks on the control 603, and the mobile phone can respond to the user operation to switch to the display interface 201 shown in FIG8a(5). The display interface 201 is the application interface of the gallery application.

[0395] Specifically, in response to the user clicking the control 603, the mobile phone can refer to the card metadata related to the image 605 to instantiate the music card 606 for the image 605 shown in Figure 8a(3) stored in the gallery application, so as to insert the music card 606 between the foreground image (the image including the foreground element 6051 described above) and the image 607 of the image 605 shown in Figure 8a(3), and the insertion position coincides with the image 607 to cover the image 607, thereby obtaining the image 605 shown in Figure 8a(5).

[0396] As shown in FIG8a(5), the instantiated music card 606 displayed on the image 605 may include, but is not limited to: an audio name 6062 (here, "Song 1"), a poster image 6061 of "Song 1", a button 6063 for switching to the previous song, a button 6064 for switching to the next song, and a button 6065 for controlling the play and pause of the audio. Unlike the various play control buttons shown in the image 607 shown in FIG8a(3), which are not interactive, the play control buttons in the music card 606 shown in FIG8a(5), such as button 6063, button 6064, and button 6065, are all interactive.

[0397] In this way, the embodiment of the present application can determine the service card to be added to the first image based on the real-time scenario, so that the type of service card added to the image and the card information can match the real-time scenario.

[0398] In addition, in the embodiment of FIG8a , the mobile phone can automatically add corresponding service cards to the first image based on the real-time context. In this case, the gallery application or camera application can provide a switch to determine whether to automatically add service cards to images. When the switch is on, the mobile phone can automatically add service cards to the first image currently captured or selected by the user based on the real-time context. When the switch is off, the mobile phone can add service cards to the first image based on user operation.

[0399] It should be understood that the real-time scenario described in the present application is not limited to the real-time scenario when the image is captured, but can also be other real-time scenarios related to adding a service card to the first image.

[0400] For example, a mobile phone displays multiple windows, including window 1 displaying a video playback interface and window 2 displaying an interface for adding a service card to a first image. The real-time scenario may be information about a video being played in the video playback window in window 1. Based on the real-time scenario, the mobile phone may add a video card related to the video being played to the first image to generate and save a corresponding target image, and then instantiate the corresponding video card for the target image when displaying the target image.

[0401] For example, if a user switches the phone's display interface from a video playback application interface to an application interface for adding a service card to a first image, the real-time context can be information about the relevant function (here, the video playback function) used by the phone's most recently running application before adding the service card to the first image, such as information about recently played videos or the user's favorite videos. This is not limited to specific information and may vary depending on the application scenario. Based on this real-time context, the phone can add video cards related to the user's favorite videos to the first image to generate and save the corresponding target image. When the target image is displayed, the corresponding video card is instantiated for the target image.

[0402] FIG8b exemplarily shows the interactive scene of the music card 606 in the image 605 shown in FIG8a(5).

[0403] As shown in FIG8b(1), the user clicks on the control 6065 in the music card 606 displayed on the image 605. The mobile phone can respond to the user operation and call the music application through the music card 606 to play the audio named "Song 1". For example, according to the playback progress recorded in the card metadata as mentioned above, the mobile phone can start playing the audio named "Song 1" from the playback progress. As shown in FIG8b(2), the mobile phone can update the graphic of the button 6065 in the music card 606 from the triangle shown in FIG8b(1) to the double vertical line shown in FIG8b(1) to indicate that the audio is in the playback state and can be paused.

[0404] In this way, the service card inserted in the image of the present application is an interactive service card, rather than a non-interactive picture showing card information, thereby achieving the effect of embedding the service card in the image.

[0405] Application Scenario 4

[0406] The difference between this application scenario 4 and the above-mentioned application scenarios 1 to 3 is that the added service card is a map card. The other processes can adopt the principles of the implementation process of any one or more application scenarios in the above-mentioned application scenarios 1 to 3, which will not be repeated here.

[0407] In this application scenario 4, FIG. 9 will be taken as an example to describe the implementation process of FIG. 3 a to FIG. 3 b above.

[0408] The following describes the application scenario 4 shown in FIG9 in conjunction with FIG3a and FIG3b, focusing on the differences between the application scenario 4 and the above-mentioned application scenarios 1 to 3, and omitting the similarities.

[0409] Please refer to Figure 9. As shown in Figure 9 (1), the display interface 201 of the mobile phone is the application interface of the gallery application. The display interface 201 displays an image 701 stored in the gallery application. The image 701 may include a foreground element 7011 (a building), a foreground element 7012 (another building) and a background element 7013 (here, environmental elements such as roads around the above two buildings).

[0410] The user can select the service card to be added to the image 701 from the service cards provided by the system through the process shown in Figure 5a above, or can select the service card to be added to the image 701 from the service cards recommended by the system that match the image scene of the image 701 through the process shown in Figure 7a. The mobile phone can also automatically determine the service card to be added to the image 701 based on the real-time scenario as described in the relevant embodiment of Figure 8a.

[0411] For example, if a user performs a long press operation on the image 701 shown in FIG9 (1), and the long press operation indicates adding a service card to the image 701, the mobile phone can determine a map card that matches the scene of the image 701 (here, a building scene) in response to the user operation. Then, the mobile phone can determine the actual location of the screen displayed by the image 701 by analyzing the shooting location of the image 701 or the foreground elements 7011 and 7012 in the image 701, thereby determining the actual location of the screen displayed by the image 701, here, "XX Building". Then, the mobile phone can call the map application and pass the actual location as a parameter to the map application to create a map card related to the actual location. Then, the mobile phone can convert the map card into an image 702 in a picture format and add it to the image 701 to obtain the image 704 (a target image) shown in FIG9 (2) and save it in the gallery application, and save the card metadata related to the image 702 to the metadata of the image 704.

[0412] As shown in FIG9 (2), image 702 displays an icon 7021, and “XX Building” is displayed near icon 7021 to indicate the actual location of the building in image 704. Image 702 also displays map information such as roads around “XX Building”.

[0413] The mobile phone can instantiate a map card for the image 704 shown in FIG9 (2), and add a map card 705 having the same card information as the image information displayed in the image 702 between the foreground element and the background element of the image 704, so as to switch from the display interface 201 shown in FIG9 (2) to the display interface 201 shown in FIG9 (3). As shown in FIG9 (3), the map card 705 is partially obscured by the foreground element 7011 and the foreground element 7012, and the map card 705 can obscure the background element 7013.

[0414] Next, the user clicks on the map card 705 shown in FIG9 (3). The mobile phone can respond to the user operation by calling the map application through the map card 705 to switch to display the display interface 703 shown in FIG9 (4). The display interface 703 is a map interface of the "XX Building" provided by the map application. The display interface 703 may include but is not limited to: a control 7032 for searching for a location, and a map screen 7031 of the "XX Building" for locating the location. The map screen 7031 can perform operations such as zooming in and out of the map information and moving the map information. This application does not limit the specific information displayed on the display interface 703. It depends on the function of the map application and is not limited here.

[0415] In this way, the mobile phone of the present application can add a map card that matches the location of the building to a picture including a building. The map card can display card information related to the location, and the map card displayed on the picture is interactive. Users can operate the map card displayed in the picture to enter the map application and perform related operations.

[0416] Example 2

[0417] In the above-mentioned Example 1 and its above-mentioned Application Scenarios 1 to 4, the process of the method for adding a service card of the present application may include: the mobile phone first adds a service card in a picture format to the first image to obtain a target image, and when displaying the target image, instantiates a service card at the position where the service card in the picture format is displayed in the target image to obtain a target image with an interactive service card.

[0418] Different from the above-mentioned Example 1 and Application Scenarios 1 to 4 of Example 1, in Example 2, the electronic device may add a target application to the application layer shown in Figure 2, and the target application may execute the image processing method to add the instantiated service card directly to the selected picture. In this way, when the target application displays the picture, an interactive service card may be displayed in the picture.

[0419] The following describes the implementation process of the method in Example 2 of the present application in conjunction with Application Scenario 5.

[0420] FIG4 exemplarily shows a schematic diagram of the implementation process of the method of Example 2.

[0421] The following describes the process of adding a service card shown in FIG. 4 of the present application in conjunction with the application scenario 5 shown in FIG. 10 a and FIG. 10 b .

[0422] In addition, the main difference between the process shown in FIG. 4 and the process shown in FIG. 3 a in the above example 1 is that there is no need to add a service card in picture format to the first image, and the principles of other processes are similar.

[0423] As shown in Figure 4, the process may include the following steps:

[0424] S500a: The mobile phone determines a first image.

[0425] The implementation principle of S500a is the same as that of S100a shown in FIG. 3a , and will not be described in detail here.

[0426] In application scenario 5, please refer to Figure 10a. The display interface 201 shown in Figure 10a (1) is the application interface of the gallery application. The display interface 201 displays an image 801 saved in the gallery application. The image 801 is a picture of a user sitting on a sofa and watching a video displayed on a screen. The image 801 exemplarily shows a foreground element 8011 (human element, sofa element), a foreground element 8012 (such as a cabinet element), and a background element 8013 (screen element). The screen element is tilted at an angle, so that the background element 8013 has a perspective parameter.

[0427] In this application scenario 5, the user can long press the image 801 shown in Figure 10a(1) to request to add a service card to the image 801. In response to the user operation, the mobile phone can instantiate a video card on the image 801 by executing the following S500b, S501a, S501c, S502, S503, and S504 to generate a target application and display the application icon 902 of the target application on the desktop interface 901 as shown in Figure 10a(2).

[0428] In a specific implementation, the mobile phone may respond to the user's long press operation on the image 801 shown in FIG10a(1) by calling the mobile phone's compiler through the gallery application to compile the code related to the process of FIG4 to generate a target application, which may include a target image with an interactive service card and the card metadata of the service card.

[0429] In some embodiments, the mobile phone can compile the above code using multiple compilers (or one compiler) associated with different operating systems to obtain multiple target applications that can run on different operating systems. The operating systems may include but are not limited to Android, iOS, Windows, Hongmeng, etc.

[0430] In this way, when a user needs to share a target application with interactive service cards across operating systems, for example, if phone 1 is running Hongmeng OS and phone 2 is running Android OS, phone 1 can generate target application 1 that can run on Hongmeng OS and target application 2 that can run on Android OS through the process shown in Figure 4. Phone 1 can run target application 1 to display the target image with interactive service cards, and phone 1 can share target application 2 to phone 2, so that phone 2 can run target application 2 on Android OS to display the target image with interactive service cards. In this way, mobile phones can achieve cross-platform sharing of the target application.

[0431] In some embodiments, when mobile phone 1's computing power is limited and it is unable to compile the target application, mobile phone 1 can interact with the cloud to enable the cloud to compile multiple target applications that can run on different operating systems, and then distribute these multiple target applications to various electronic devices. For example, the cloud can send target application 1 to mobile phone 1 and target application 2 to mobile phone 2, thereby enabling mobile phone 1 to share the target applications with mobile phone 2.

[0432] Since the target application 2 shared by mobile phone 1 to mobile phone 2 includes a target image with an interactive service card, mobile phone 2 can display the target image with the instantiated service card after starting the target application 2, and can interact with the service card through the target application.

[0433] The application scenario of sharing the target application is similar to the application scenario of the embodiments of Figures 5c and 5d. The only difference is that the shared content is not an image but a target application. The specific process will not be repeated here.

[0434] S500b: The mobile phone determines the first service card.

[0435] The implementation principle of S500b is the same as that of S100b shown in FIG. 3a , and will not be described in detail here.

[0436] In the application scenario 5 shown in FIG10a , the mobile phone can determine that the first service card matching the image 801 shown in FIG10a (1) is a video card, and the mobile phone can identify the background element 8013 in the image 801 to determine the video name, video playback progress and other information of the video watched by the person in the image 801; then, the mobile phone can call the interface of the video application and pass the above information as a parameter to the interface to create the video card 803 shown in FIG10b .

[0437] As shown in FIG10 b , the video card 803 may include but is not limited to: a video screen 8033 , a play pause control 8031 ​​, a progress bar 8034 , a play progress adjustment control 8032 and other card information.

[0438] Among them, the video screen 8033 and the video screen shown by the background element 8013 shown in Figure 10a (1) correspond to the same frame of the same video. In this way, the video card 803 created by this application is a video card that matches the video screen displayed by the image 801 on the screen.

[0439] As shown in FIG. 4 , after S500 a , the method may further include S501 a .

[0440] S501a: The mobile phone may divide the first image into a foreground image including foreground elements and a background image including background elements.

[0441] The implementation principle of S501a is the same as that of S101a shown in FIG. 3a , and will not be described in detail here.

[0442] In application scenario 5 shown in Figure 10a, the mobile phone can divide image 801 (an example of the first image) into a foreground image including foreground elements 8011 and foreground elements 8012, and other areas are transparent areas, and a background image including background elements 8013, and other areas are transparent areas.

[0443] Optionally, as shown in FIG4 , after S500b , the method further includes S501c .

[0444] S501c, the mobile phone can adjust the first service card to the second service card.

[0445] The implementation principle of S501c is the same as that of S101c shown in FIG. 3a , with the only difference being that the object of adjustment is changed from an image to a service card.

[0446] In some embodiments, the mobile phone can determine a first area in the first image where a service card is to be inserted; and the mobile phone can refer to image parameters of the first image related to the first area to adjust the corresponding image parameters of the first service card, so that after the second service card with adjusted image parameters is inserted into the first image, the second service card can be more integrated with the picture of the first image. In this way, the service card inserted into the first image can be displayed without being abrupt on the entire image.

[0447] In some embodiments, as described above, the first image of the service card to be added can be divided into a foreground image and a background image. Then, when the mobile phone determines the first area to which the service card is to be inserted, the corresponding service card can be inserted between the foreground image and the background image, and the foreground element in the foreground image can block the area ratio of the service card within a preset threshold range, for example, 3% to 20% of the area of ​​the service card is blocked by the foreground element, so as to determine the position of the first area. The preset threshold range can be flexibly changed according to actual needs and is not limited here.

[0448] In some embodiments, the mobile phone may also determine the first area based on a location or area selected by the user in the first image. For example, if the user selects the coordinates of a service card to be added in the first image, the mobile phone may determine the first area including the coordinates based on the coordinates.

[0449] In some embodiments, the image parameters of the above-mentioned first image related to the first area may include but are not limited to: the size of the first area, the shape of the first area, the perspective parameters of the background elements located in the first area (which may include at least one of the perspective direction and the perspective angle), the color, hue, transparency, lighting characteristics and other information of the foreground elements and / or background elements related to the first area.

[0450] Among them, the solution for the mobile phone to adjust the first service card by referring to the perspective parameters (which may include at least one of the perspective direction and perspective angle) of the background elements located in the first area will be explained in detail in the application scenario 5 of Figures 10a and 10b below, and will not be repeated here.

[0451] The lighting characteristics may be at least one lighting characteristic such as direct lighting, reflected lighting, and shadow.

[0452] For example, the background image of the first image includes a sun element, and the light emitted by the sun element shines on the human element in the foreground image. The insertion position of the first service card in the first image (such as the first area mentioned above) is on the illumination path. Then, the mobile phone can use ray tracing (RT) technology to calculate the direct illumination characteristics of the area or position on the illumination path in the first service card, and render the direct illumination characteristics of the position on the illumination path in the first service card to obtain a second service card, so that after the second service card is inserted into the first area in the first image, a target image is obtained. The position on the illumination path in the second service card displayed on the target image can display a visual effect of direct illumination. Optionally, since calculating the illumination characteristics requires a high computing power, the mobile phone can interact with the server to implement the calculation of the direct illumination characteristics of the image to obtain the rendering result of the second service card, and then use the rendering result to obtain the second service card.

[0453] Of course, the way in which the mobile phone renders a visual effect with corresponding lighting characteristics on the service card is not limited to the above example, and other technologies such as AIGC can also be used to calculate the lighting characteristics of the service card.

[0454] In application scenario 5 shown in Figures 10a and 10b, the mobile phone can determine that the first area where the service card is to be inserted is the area where the background element 8013 shown in Figure 10a (1) is located. In this embodiment, the background element 8013 located in the first area is a curtain, and the curtain has a tilt angle, so that the background element 8013 has a perspective parameter.

[0455] As shown in FIG10b , the mobile phone can adjust the perspective parameters of video card 803 (an example of a first service card) based on the perspective parameters of background element 8013 through S501b shown in FIG4 , specifically adjusting the perspective direction and perspective angle, to obtain video card 802. Video card 802 may include, but is not limited to, card information such as a video screen 8023, a play / pause control 8021, a progress bar 8024, and a play / pause control 8022.

[0456] Because the entire video card 803 changes to video card 802 through adjustments to its perspective direction and angle, there are changes in perspective parameters between video screen 8033 and video screen 8023, between play / pause controls 8031 ​​and play / pause controls 8021, between progress bars 8034 and progress bars 8024, and between play / progress control controls 8032 and play / progress control controls 8022. Furthermore, the changes in perspective parameters between video card 803 and video card 802 also cause the card shapes to change.

[0457] S502: The mobile phone combines a foreground image including foreground elements, a background image including background elements, and a second service card into a target image.

[0458] The implementation principle of S502 is the same as that of S102 shown in FIG. 3 a , and will not be described in detail here.

[0459] As shown in FIG4 , when synthesizing, the mobile phone uses the foreground image including the foreground elements as the upper layer, the second service card as the middle layer, and the background image including the background elements as the lower layer to synthesize the target image.

[0460] In application scenario 5 shown in Figures 10a and 10b, the mobile phone can insert the video card 802 (an example of the second service card) shown in Figure 10b into the first area between the foreground image and the background image of the image 801 shown in Figure 10a(1), where the first area is the position corresponding to the background element 8013, so that the video card 802 can cover the background element 8013 to obtain the target image 801 with the interactive video card 802 as shown in Figure 10a(3).

[0461] Optionally, in S503 , the mobile phone may determine card metadata related to the target image based on the target image, the first service card, and the second service card.

[0462] The implementation principle of S502 is the same as that of S102 shown in FIG. 3 a , and will not be described in detail here.

[0463] The card metadata may include but is not limited to at least one of the following: card type, card information, card size (such as the size of the first area mentioned above), card position (such as the position of the first area in the first image), card shape (such as the shape of the first area mentioned above), and card image parameters.

[0464] The image parameters of the card are the image parameters of the second service card after the first service card is adjusted to the second service card through the above S501c, or the image parameters of the card can also be the image adjustment parameters (such as size adjustment parameters, transparency adjustment parameters, etc.) used when the first service card is adjusted to the second service card through the above S501c, wherein the image adjustment parameters can indicate information such as the adjustment direction and adjustment amount of the image parameters. The image adjustment parameters are relative to the above image parameters. The image adjustment parameters are relative image parameters and absolute image parameters.

[0465] In some embodiments, the image parameters may include, but are not limited to, information such as color, hue, transparency, lighting characteristics, and perspective parameters.

[0466] For example, the mobile phone can determine the card type and card information of the first service card based on the first service card determined in S500b, and the mobile phone can determine the card perspective parameters, card size, card position, card color, card tone, card lighting characteristics, card transparency, etc. in the card metadata based on the perspective parameters, size and color, hue, lighting characteristics, transparency, etc. of the synthesis position (the above-mentioned first area) in the first image of the second service card used when performing the synthesis operation in S502.

[0467] After S502 and S503 , as shown in FIG. 4 , the mobile phone may execute S504 .

[0468] S504: The mobile phone may generate a target application, where the target application includes a target image with an interactive service card, and optionally includes card metadata of the service card.

[0469] As shown in FIG10a , in response to a user long pressing the image 801 shown in FIG10a (1), the mobile phone can compile the above-mentioned processes S500b, S501a, S501c, S502, S503, and S504 to generate a target application, and display the application icon 902 of the target application on the desktop interface 901 shown in FIG10a (2).

[0470] When the user clicks the application icon 902 shown in Figure 10a(2), the mobile phone can respond to the user operation and start the target application to switch to display the display interface 903 shown in Figure 10a(3). The display interface 903 is the application interface of the target application, and the display interface 903 may include the image 801 with the interactive video card 802 stored in the target application obtained in S502, as well as the control 207, etc.

[0471] Among them, there are certain differences between the control 207 here and the control 207 involved in the above-mentioned application scenarios 1 to application scenarios 4. The difference is that the control 207 shown in Figure 10a in this application scenario 5 is used to indicate the sharing target application, rather than inserting an image of a service card in picture format. The other process principles are the same and will not be repeated here.

[0472] In this embodiment, the mobile phone can instantiate a service card on the selected image without first pasting a picture of a service card on the image and then pasting the instantiated service card, which simplifies the operation process of adding a service card to the image.

[0473] Continuing with FIG10a(3), the target application generated by the mobile phone may include an image 801 with a video card 802, and the video card 802 is interactive. As shown in FIG10a(3), when the user clicks the play / pause control 8021, the mobile phone can respond to the user operation by calling the video application through the video card 802 to start playing the video displayed by the video card 802, thereby switching to display interface 903 as shown in FIG10a(4).

[0474] As shown in FIG10a(4), the play / pause control 8021 in the video card 802 displayed on the display interface 903 is switched from the triangle shown in FIG10a(2) to the double vertical lines shown in FIG10a(3), and the switched play / pause control 8021 still has the adjusted perspective parameters. As shown in FIG10a(3), the double vertical lines shown in the play / pause control 8021 are tilted, while the double vertical lines are vertical. In this way, the video card 802 shown in FIG10a(3) can play the video according to the recorded video playback progress to display the video playback screen.

[0475] Continuing with reference to FIG10a(4), the user slides the playback progress adjustment control 8022 in the video card 802 from position A indicated by the dotted arrow on the progress bar 8024 to position B indicated by the dotted arrow with a finger to change the video playback progress of the video played by the video card 802. In response to the user operation, the mobile phone can call the video application through the video card 802 to adjust the playback progress of the video card 802 and switch the video screen displayed by the video card 802 to switch to the video card 802 as shown in FIG10a(5).

[0476] As shown in FIG8a(5), the playback progress adjustment control 8022 is located at position B indicated by the dotted arrow, indicating that the playback progress of the video played by the video card 802 has been adjusted to the playback progress indicated by the position B, and the video screen 8023 displayed by the video card 802 has also switched from the video screen 8023 shown in FIG8a(4) to the video screen related to the adjusted playback progress.

[0477] In this embodiment, a target image with service cards can be obtained by generating a target application, and the service cards in the target image are interactive.

[0478] It should be understood that in the above Examples 1 and 2, the technical solution of the method of the present application is illustrated by instantiating a service card in an image. When multiple service cards are instantiated on an image, the implementation principle is the same and will not be repeated here.

[0479] Additionally, in Example 2 above, the mobile phone can instantiate at least one service card for a single first image to generate a target application, which can include the target image with the at least one service card. If the mobile phone needs to use the method of Example 2 to obtain multiple target images with instantiated service cards, the mobile phone can perform the process shown in Figure 4 for each first image to obtain multiple target applications, each of which can include a target image with an instantiated service card, where each target image corresponds to a first image.

[0480] This Example 2 mainly introduces the differences between the above Example 1. The similarities between the two examples are not introduced in Example 2. For details, please refer to the introduction of the relevant embodiments of Example 1, and no further description is given here.

[0481] It should be understood that the method of the present application can insert interactive service cards into images, but the types of service cards are not limited to the weather cards, music cards, video cards, map cards, and sports cards listed in the above application scenarios. It can also be any type of service card such as time cards, picture cards, calendar cards, schedule cards, to-do cards, payment cards, search cards, etc. The implementation principles are the same and will not be elaborated here.

[0482] In the above example 1 and its above application scenarios 1 to 4, and the above example 2 and its application scenario 5, the UI control added to the picture as a service card is taken as an example to illustrate the implementation process of the method of the present application.

[0483] In some embodiments, the UI control may also be a notification message in a notification bar interface or a live window in a notification bar interface. The mobile phone may receive a user operation on the notification message or live window, where the operation is used to instruct the operated notification message or live window to be added to the image. The mobile phone may respond to the user operation by instantiating the notification message or live window in the corresponding first area of ​​the first image using the relevant method of Example 1 or Example 2 to obtain a target image, so that the interactive notification message or live window can be displayed on the target image.

[0484] For example, the interactive scenario of the notification message can be: the user can click on the notification message displayed on the target image to display the notification bar interface on the mobile phone, or the mobile phone displays the application interface of the application that pushed the notification message, which application interface includes application information related to the notification message.

[0485] For example, the interactive scenario of the live window may be: the user clicks on the live window displayed on the target image, and the mobile phone can open the corresponding application interface related to the live window in response to the user operation.

[0486] It should be understood that only two interaction methods are listed here as examples. This application does not limit the interaction method and its scenario. The specific interaction method depends on the interactive function of the notification message or the live window itself.

[0487] For example, the interactive scenario of a notification message can also be: in response to the user's long press on the notification message displayed on the target image (the operation indicates that the notification message is converted into a service card), the mobile phone converts the notification message displayed on the target image into an instantiated service card. The card information of the service card is related to the message content of the notification message, for example, it can be some structured notification information. For example, if the notification message is a pickup notification, the card information may include information such as the pickup code, pickup address, and product type. In this way, the interactive control displayed on the target image can be refreshed from the notification message to the service card, and the user can continue to interact with the service card on the target image.

[0488] In some embodiments, the UI control may also be an interactive text box (such as a search box, etc.), an interactive button, etc.

[0489] For example, if the UI control is a search box in a map application, using Example 1 as an example, the phone can convert the search box into an image and add it to the first image to obtain the target image. Then, each time the target image is displayed, the phone can display the first area of ​​the search box in image format within the target image and call the map application to create a search box control, thereby instantiating the map application's search box on the target image. In this way, the user can enter information such as a destination in the instantiated search box displayed on the target image and trigger a search operation. The phone can then call the map application to search for the destination, displaying the map application interface with search results for the destination.

[0490] For example, the UI control includes the text of a phone number and a dial button for initiating a voice call. Taking Example 1 as an example, a user operates on the phone number of a contact in the address book, instructing to add the text of the phone number and the dial button for initiating a voice call to the first image. In response to the user operation, the phone can convert the phone number text and the dial button into images and combine them into a single image. The combined image includes the phone number information and the dial button icon. The phone can then add the combined image to the first image to obtain the target image. When the phone displays the target image, the phone can instantiate a dial button of the phone application at the location where the dial button icon is displayed on the target image. Then, the user clicks the instantiated dial button displayed on the target image. In response to the user operation, the phone can identify the phone number information displayed on the target image, call the dial interface of the phone application, and pass the phone number information as a parameter to the interface to implement the interactive function of calling the specific contact corresponding to the specific phone number displayed in the photo. Of course, the text of the phone number in the above UI control can also be a picture of the avatar of the corresponding contact. The implementation principle is similar and will not be repeated here.

[0491] In some embodiments, the UI control may also be a floating control (such as a floating window, a floating ball, etc.) or an application interface (such as a chat interface, etc., without limitation).

[0492] Taking the solution of Example 2 as an example, the user can operate the floating control or application interface (such as a chat interface) of interest. The operation indicates that the corresponding floating control or chat interface will be added to the first image. In response to the user operation, the mobile phone can instantiate the floating control or the corresponding chat interface at the corresponding position of the first image to obtain the target application. The target application may include a target image displaying the interactive floating control or chat interface. When the user opens the target application, the mobile phone may display the application interface of the target application, which displays the target image with the interactive floating control or the interactive chat interface.

[0493] For example, by clicking a floating control (such as a floating ball) displayed on the target image, the mobile phone can display the application interface (such as a video playback interface) corresponding to the floating ball. The video playback interface can be displayed in the application interface of the target application, or it can be displayed in the video application to which the video playback interface belongs (that is, the mobile phone can display the video playback interface of the video application). There is no restriction here.

[0494] For another example, the user inputs into the chat interface displayed on the target image, and the mobile phone can respond to the user operation by calling the corresponding chat application to send the input chat content.

[0495] It should be understood that the above only illustrates some of the embodiments of the method of the present application to add UI controls other than service cards to pictures. The principles and methods of the process of adding UI controls other than service cards to pictures in the present application can refer to the relevant descriptions of the embodiments of Example 1 and Example 2 above, and will not be repeated here one by one.

[0496] It should be noted that the same figure marks in the above-mentioned Figures 3a to 10b represent the same objects. Therefore, the figure marks of each figure are not explained one by one. The figure marks not mentioned in the above-mentioned figures can refer to the explanations of the same figure marks mentioned in Figures 3a to 10b, and will not be repeated here.

[0497] It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0498] In one possible implementation, the present application further provides an image processing device. The image processing device includes: a first display module configured to display a target image in response to a first user operation received on a first image, wherein the target image is a first image displaying a target user interface (UI) control; and an output module configured to output an interaction result related to the target UI control in response to a second user operation received on the target UI control.

[0499] In a possible implementation, the target UI control is a music control, and the music control is related to a music application running in the background when the first user operation is received.

[0500] In a possible implementation, the target UI control is related to the first image.

[0501] In a possible implementation, the control type of the target UI control is related to the image content of the first image.

[0502] In a possible implementation, the control information displayed by the target UI control is related to the image content of the first image.

[0503] In a possible implementation, the perspective parameter of the target UI control is the same as the perspective parameter of the target image element corresponding to the display position of the target UI control in the first image.

[0504] In a possible embodiment, the first display module is specifically used to: display a second image, wherein the second image is the first image including a first UI element, wherein the first UI element is a first UI control in a picture format; display a target UI control related to the first UI control on the second image to display a target image, wherein the first UI element is covered by the target UI control.

[0505] In a possible implementation, the first UI control and the target UI control each display the same control information.

[0506] In a possible implementation, the image processing device further includes: a storage module for storing the first control metadata of the first UI control in association with the second image; and a generation module for generating the target UI control based on the first control metadata.

[0507] In a possible implementation, the image processing apparatus further includes: a refresh module, configured to refresh the first control metadata to the target control metadata of the target UI control.

[0508] In a possible implementation, the image processing apparatus further includes: a second display module configured to display a target UI control related to the first UI control on the second image in response to a received display operation on the second image, so as to display the target image.

[0509] In a possible implementation, the image processing apparatus further includes: a sending module configured to send the second image and the first control metadata associated with the second image in response to a received sharing operation on the second image.

[0510] The first display module is specifically configured to: generate a target application; and display a target image in response to a received start-up operation on the target application.

[0511] The above-mentioned image processing device can be an electronic device, a chip, etc., which is not limited here.

[0512] The effects of the above-mentioned image processing device are the same as the relevant effects of the above-mentioned method embodiment, and will not be repeated here.

[0513] In one example, FIG11 shows a schematic block diagram of an apparatus 300 according to an embodiment of the present application. The apparatus 300 may include: a processor 301 , optionally a transceiver / transceiver pin 302 , and optionally a memory 303 .

[0514] The various components of the device 300 are coupled together via a bus 304, wherein the bus 304 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the various buses are collectively referred to as bus 304 in the figure.

[0515] Optionally, the memory 303 may be used for instructions in the aforementioned method embodiment. The processor 301 may be used to execute instructions in the memory 303 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.

[0516] The apparatus 300 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0517] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0518] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

[0519] This embodiment further provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0520] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the image processing method in the above-mentioned method embodiments.

[0521] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0522] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0523] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0524] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0525] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0526] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0527] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0528] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0529] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in hardware or by executing software instructions by a processor. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device. Of course, the processor and storage medium can also be present in a network device as discrete components.

[0530] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0531] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: In response to receiving a first user operation on a first image, displaying a target image, wherein the target image is a first image that displays a target user interface UI control; In response to receiving a second user operation on the target UI control, an interaction result related to the target UI control is output.

2. The method according to claim 1, characterized in that: The target UI control is a music control, and the music control is related to a music application running in the background when the first user operation is received.

3. The method according to claim 1, characterized in that The control type of the target UI control is related to the image content of the first image.

4. The method according to any one of claims 1 to 3, characterized in that The control information displayed by the target UI control is related to the image content of the first image.

5. The method according to any one of claims 1 to 4, characterized in that: The perspective parameter of the target UI control is the same as the perspective parameter of the target image element in the first image corresponding to the display position of the target UI control.

6. The method according to any one of claims 1 to 5, characterized in that The display target image comprises: Display a second image, wherein the second image is the first image including a first UI element, wherein the first UI element is a first UI control in a picture format; A target UI control related to the first UI control is displayed on the second image to display a target image, wherein the first UI element is covered by the target UI control.

7. The method according to claim 6, characterized in that The first UI control and the target UI control each display the same control information.

8. The method according to claim 6 or 7, characterized in that: The method further comprises: storing the first control metadata of the first UI control in association with the second image; The target UI control is generated based on the first control metadata.

9. The method according to claim 8, characterized in that The method further comprises: The first control metadata is refreshed as the target control metadata of the target UI control.

10. The method according to any one of claims 6 to 9, characterized in that The method further comprises: In response to the received display operation on the second image, a target UI control related to the first UI control is displayed on the second image to display the target image.

11. The method according to any one of claims 8 to 10, characterized in that The method further comprises: In response to the received sharing operation on the second image, the second image and the first control metadata associated with the second image are sent.

12. The method according to any one of claims 1 to 5, characterized in that The display target image comprises: Generate target application; In response to the received launch operation on the target application, a target image is displayed.

13. An electronic device, characterized in that: The electronic device comprises: A first display module, configured to display a target image in response to a received first user operation on a first image, wherein the target image is a first image that displays a target user interface UI control; An output module is used to output an interaction result related to the target UI control in response to a received second user operation on the target UI control.

14. An electronic device, characterized in that: include: a memory and a processor, the memory and the processor being coupled; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the image processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The invention comprises a computer program, which, when being executed on an electronic device, enables the electronic device to execute the image processing method according to any one of claims 1 to 12.

16. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the image processing method according to any one of claims 1 to 12.

17. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the image processing method described in any one of claims 1 to 12.

Images