Image processing method and electronic equipment

Abstract

The embodiment of the invention provides an image processing method and electronic equipment, and relates to the technical field of terminal equipment, and the method comprises the steps: displaying an interactive UI control on an image based on user operation, and enabling the function and service of the UI control to be embedded in the image for a user to use.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of terminal devices, and in particular to an image processing method and electronic device. Background Art

[0002] With the development of technology, electronic devices can provide UI controls such as service cards to facilitate users to use corresponding functions or services of applications or electronic device systems.

[0003] Taking a mobile phone as an example, in some scenarios, a user desires to add a UI control (such as a service card) supported by the mobile phone to a picture for display. In the prior art, the user can display a screen with a service card in the picture by overlaying, cropping, and erasing a screenshot of the service card and the picture to be added with the card.

[0004] However, in the prior art, the pictures with service cards added thereto are only static images of the service cards, and are unable to interact to provide the functions of the service cards. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides an image processing method and an electronic device. In the method, an interactive UI control can be displayed on an image, so that the functions and services of the UI control can be embedded in the image for users to use.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device. The method comprises: in response to a first user operation received on a first image, displaying a target image, wherein the target image is a first image displaying a target user interface UI control; in response to a second user operation received on the target UI control, outputting an interaction result related to the target UI control.

[0007] The first image may be an original image without any UI control added thereto, or may be an image after a first UI control in a picture format is added to the original image, etc., and there is no limitation here.

[0008] In an example, the first user operation may be an operation of adding an interactive UI control to the original image. The operation may be any form of screen touch input or a physical button press input, etc., which is not limited here.

[0009] In another example, the first image is an image with a UI control in a picture format added. When the user performs an operation to request to display the first image (for example, the first user operation is an opening operation on the first image), the electronic device can execute the above method to instantiate the target UI control on the first image.

[0010] In yet another example, the first user operation may also be a photographing operation, so as to instantiate and display the target UI control on the first photographed image.

[0011] In addition, in some examples, the first user operation may also include a selection operation on a target UI control to select a UI control to be added to the first image this time.

[0012] In addition, the above-mentioned target UI control in the first aspect may be a UI control added by the first user operation instruction, and / or may be a UI control that the electronic device automatically determines to match the first image in combination with the first image.

[0013] The target UI control can be any interactive UI control. For example, the control type of the target UI control can be a service card (also called a widget in some scenarios), where the service card is also divided into many types, such as music cards, map cards, navigation cards, video cards, sports cards, etc. The card types (i.e., control types) of various service cards are also different. The subsequent implementation method takes the UI control as a service card as an example to illustrate the method of the present application.

[0014] In addition, the target UI control can also be a notification message, a live window, a floating control (such as a floating window, a floating ball, etc.), an interactive button (such as a button for calling a contact), a text box (such as a search box), an application interface (such as a chat interface, etc., without limitation), etc.

[0015] In an embodiment of the present application, the electronic device can respond to user operations to display a target UI control on the first image, and the target UI control is an interactive UI control. By operating the target UI control, the electronic device can respond to the user's operation on the target UI control to output an interactive result related to the target UI control, so that the target UI control displayed by the target image is not a static image, but an interactive UI control. By interacting with the UI control displayed on the target image, the user can use the functions and services provided by the UI control to expand the usage scenarios of the UI control. In this way, the UI control embedded in the target image can also provide functions and services related to the UI control for user interaction.

[0016] Among them, the way of outputting the interaction result related to the target UI control may include but is not limited to: displaying the application interface, refreshing the display information of the UI control, outputting audio, etc.

[0017] For example, if the target UI control is a music card, the second user operation may be clicking the play button on the music card. In this way, when the electronic device outputs the interaction result, it may refresh the triangular icon of the play button of the music card displayed on the first image to double vertical lines to refresh the display information of the music card. In addition, it may also output the audio data of the song displayed on the music card to play the song with the title displayed on the music card.

[0018] For another example, if the target UI control is a search box, then after the user enters a search object in the search box through a second user operation and triggers a search operation, the electronic device may display an application interface of a search engine application when outputting interactive results. The application interface is a search result interface, and the search results in the search result interface are the search results of the search engine application for the search object, and the search engine application provides the above-mentioned search box.

[0019] According to the first aspect, the target UI control is related to a music application running in the background when the first user operation is received.

[0020] For example, the target UI control is a music control. The music control may be related to a music application running in the background when the electronic device receives the first user operation.

[0021] For example, when a user selects a picture in a gallery application through a first user operation to request that a service card be added to it (or when the user uses a camera application to take a picture through a first user operation), and a music application is running in the background of the electronic device (for example, music is being played using the music application), the electronic device can add music controls related to the music application to the picture.

[0022] For example, the music control may be a music card, and the card information displayed on the music card may include information about the song being played by the music application. Of course, the music control may also be other types of interactive music controls, such as a music control including a play and pause button for the music and the name of the music, or a music control that is a floating window of the music playback interface of the music application. The music control of the present application may be any interactive UI control related to the above-mentioned music application, such as a service card, a floating control, an application interface, etc.

[0023] Optionally, the control information displayed by the music control may be related to the music application.

[0024] In an embodiment of the present application, the electronic device can add interactive music controls to the first image in conjunction with the music application running in the background when the user adds UI controls to the first image, so that the target UI controls displayed on the target image can be related to the music scenario when the user adds the UI controls.

[0025] According to the first aspect, or any implementation of the first aspect above, the target UI control is related to the first image.

[0026] In some implementations, the target UI control may have target control metadata to describe the target UI control.

[0027] The target control metadata may include, but is not limited to: control type, control size, control display position, control display information, and control image parameters.

[0028] The control display information is used to describe the content displayed on the control, and the control display information may include but is not limited to at least one of: a button, an icon, an image, a text, etc.

[0029] The image parameters of the control may include, but are not limited to, at least one of perspective parameters, color, hue, transparency, and lighting characteristics.

[0030] At least one of the above-mentioned metadata in the target UI control may be related to the first image. In other words, the electronic device may determine at least one attribute of the target UI control based on the first image, wherein the above-mentioned target control metadata may also be described as an attribute of the target UI control.

[0031] In some implementations, the control type of the target UI control can be related to the image content of the first image.

[0032] The image content may be the image semantics of the first image, or the scene represented by the first image, etc.

[0033] For example, if the image content of the first image is a certain building, then its semantics is the building. Then the target UI control may be a navigation card or a map card.

[0034] For another example, if the image content of the first image is a picture of a person running in a gym, the scene indicated by the first image is a sports scene, and the target UI control may be a sports card.

[0035] In some implementations, the control display position of the target UI control may be related to the layout between the foreground elements and the background elements of the first image. For example, the display position of the target UI control on the first image is a specific position between the foreground elements and the background elements.

[0036] In some implementations, the control size of the target UI control may be related to the size, depth of field, etc. of image elements (foreground elements and / or background elements) in the first image.

[0037] For example, if the image content of the first image is a user watching a movie through a projector screen, then the target UI control added to the first image may be a video card. In addition, the size of the video card may be the same as the size of the screen in the first image.

[0038] In some implementations, the control display information of the target UI control may be related to the shooting time, shooting location, image content, etc. of the first image.

[0039] For example, if the target UI control is a weather card, the card information displayed by the weather card may be weather information related to the shooting time and shooting location of the first image.

[0040] In some implementations, the image parameters of the target UI control may be related to information such as color, hue, transparency, lighting characteristics, perspective parameters, etc. of foreground elements and / or background elements of the target area in the first image where the target control is to be displayed.

[0041] In an embodiment of the present application, the electronic device can display a target UI control related to the first image on the first image, so that the first image can match at least one of the control type, control size, control display position, control display information, and image parameters of the target UI control. In this way, the UI control added to the picture can be better matched with the picture, and the image fusion and picture harmony between the target UI control and the first image are higher.

[0042] According to the first aspect, or any implementation manner of the first aspect above, the control type of the target UI control is related to the image content of the first image.

[0043] For example, if the image content of the first image includes a building, the electronic device may add a service card related to a location, such as a map card or a navigation card, to the first image. Optionally, the map information displayed in the map card may be map information about the location of the building, and the navigation card may be navigation information for navigating to the address of the building.

[0044] For another example, the image content of the first image is a picture of a person running in a gym, and the electronic device can add a sports card to the first image.

[0045] For example, if the image content of the first image is a user holding a microphone and singing, the electronic device can add a music card to the first image.

[0046] For example, if the image content of the first image is a scene of a person making a phone call, the electronic device can add a contact card, or display a target UI control such as a button for making a call to a specific contact on the first image.

[0047] In this way, the present application can combine the image content of the first image to add a corresponding type of UI control that better matches the image content to the first image, so that the image content of the first image displaying the target UI control is more compatible with the control type of the added target UI control.

[0048] According to the first aspect, or any implementation of the first aspect above, the control information displayed by the target UI control is related to the image content of the first image.

[0049] In some implementations, the control display information of the target UI control (also expressed as displayed control information) may be related to the shooting time, shooting location, image content, etc. of the first image.

[0050] For example, if the target UI control is a weather card, the card information displayed by the weather card may be weather information related to the shooting time and shooting location of the first image.

[0051] In this way, the control display information of the interactive target UI control added by the electronic device to the first image can match the image content of the first image, so that the first image and the target UI control have a higher degree of semantic integration and matching.

[0052] According to the first aspect, or any implementation of the first aspect above, the perspective parameter of the target UI control is the same as the perspective parameter of the target image element corresponding to the display position of the target UI control in the first image.

[0053] The above display position is the position (also referred to as the first area) where the target UI control is to be displayed in the first image. If the target image pixels (foreground elements and / or background elements, depending on the display position of the target UI control) in the first area where the target UI control is to be added in the first image have a tilt angle, for example, the curtain element in the first image is not photographed perpendicular to its surface, so that the curtain element has a tilt angle when displayed on the first image; for another example, the building displayed in the first image has curved glass. No matter from which angle the building is photographed, the glass of the building displayed in the first image has a tilt angle. In such cases, the target image elements (such as building glass elements, curtain elements, etc.) in the first image have perspective parameters. When the electronic device adds the UI control to the area where the target image element is located in the first image, the UI control can cover the target image element. In order to make the tilt angle of the target image element consistent with the tilt angle of the UI control so that the two are more closely matched, the electronic device can make the perspective parameters of the target UI control added to the first image the same as the perspective parameters of the target image element. The perspective parameter may include a perspective direction and / or a perspective angle.

[0054] In this way, when the target image element of the target UI control displayed on the first image has a tilt angle, the target UI control added to the first image in the present application is also adjusted through corresponding perspective parameters, so that the tilt angle of the target UI control added to the first image can be consistent with the tilt angle of the above-mentioned target image element. In this way, the target UI control displayed on the first image can be more closely aligned with the target image element at the corresponding display position, and the target UI control can be more integrated with the image elements around the target image element.

[0055] According to the first aspect, or any implementation manner of the first aspect above, displaying the target image includes: displaying a second image, wherein the second image is the first image including a first UI element, wherein the first UI element is a first UI control in a picture format; displaying a target UI control related to the first UI control on the second image to display the target image, wherein the first UI element is covered by the target UI control.

[0056] In this implementation, the first image is an original image without UI elements added thereto, so when the user requests to add UI elements thereto through a first user operation, the electronic device of the present application may first add a first UI control in a picture format to the first image to obtain a second image; then, when the electronic device displays the target UI control on the second image, it may be triggered to display the target UI control on the second image in response to another received user operation, or it may be automatically after displaying the second image, automatically displaying the corresponding position of the first UI control in a picture format in the second image (because it is in a picture format, it is not interactive), to instantiate and display the target UI control, so that the first UI control in a picture format may be covered by the target UI control. Of course, the display range of the target UI control on the second image may also exceed the display range of the first UI control in a picture format in the second image, so as to achieve the effect that the target UI control covers the first UI control.

[0057] In some embodiments, the display area corresponding to the target UI control on the first image includes the display area corresponding to the first UI element on the first image to ensure that the interactive target UI control can cover the first UI element in a static picture format in the first image.

[0058] In addition, the instantiated target UI control can be the same control type as the above-mentioned first UI control, or the control display information of the target UI control and the first UI control can be the same, or the target UI control and the first UI control are the same UI control. There is no limitation here. As long as the target UI control is related to the first UI control, the correlation between the two can be the correlation of any type of metadata in the target control metadata mentioned in the above implementation method.

[0059] In one example, the electronic device may generate a target UI control (also referred to as an instantiated target UI control) based on the first UI element in the first image, and display the target UI control at a corresponding position in the second image.

[0060] For example, first control metadata about a first UI control of a first UI element may be carried in metadata of a first image, and the electronic device may generate a target UI control related to the first UI control based on the first control metadata.

[0061] For another example, the electronic device may perform image recognition on the second image to determine first control metadata of a first UI control in a picture format embedded in the second image, and generate a target UI control related to the first UI control in combination with the first control metadata.

[0062] In addition, in some embodiments, when the electronic device generates a second image based on a first image, or generates the above-mentioned target image based on the first image or the second image, the first image can be divided into a foreground image where the foreground elements are located and a background image where the background elements are located, wherein the foreground image can be one or more images, and the background image can be one or more images.

[0063] Among them, the foreground image is an image that only includes the foreground element (the area other than the foreground element is a transparent area), and the background image is an image that only includes the background element obtained by segmenting from the first image. The sizes of the foreground image and the background image are the same as those of the first image, the second image, and the target image.

[0064] In some embodiments, the first UI control in the above picture format, the foreground image obtained by segmenting the first image, and the background image obtained by segmenting the first image can be used as three layers from top to bottom in sequence to synthesize the second image.

[0065] Alternatively, in some embodiments, the foreground image obtained by segmenting the first image, the first UI control in the picture format, and the background image obtained by segmenting the first image can be sequentially used as three layers from top to bottom to synthesize the second image. Optionally, in the second image, the first UI control in the picture format can be partially obscured by the foreground element in the foreground image.

[0066] In some embodiments, when the electronic device does not generate the second image but directly uses the first image to generate the target image, the foreground image obtained by segmenting the first image, the target UI control, and the background image obtained by segmenting the first image can be sequentially used as three layers from top to bottom to synthesize the target image. Alternatively, the target UI control can also be used as the topmost image.

[0067] In some embodiments, when the electronic device generates a target image based on the second image, the layer where the target UI control is located may be located above the layer where the first UI control in the image format is located, so as to achieve the shielding of the first UI control by the target UI control.

[0068] In this implementation, the electronic device may embed a first UI control in a picture format on a first image to display a second image with the first UI control in a picture format, and then, based on the display position of the first UI control in the picture format in the second image, instantiate a target UI control (for example, create a service card at the display position) to obtain a target image with an interactive target UI control. For example, the electronic device may save the second image, and subsequently, when the second image is displayed, instantiate the target UI control at the corresponding display position to make the target UI control displayed on the target image interactive.

[0069] According to the first aspect, or any implementation of the first aspect above, the control information displayed by the first UI control and the target UI control respectively (ie, the control display information mentioned in the control metadata above) is the same.

[0070] In some embodiments, the first UI control and the target UI control may be the same control. In this way, the instantiation process of the target UI control, that is, the process in which the image displayed by the electronic device changes from the second image to the target image, is indistinguishable to the user by the naked eye. That is, there is no difference in display effect between the first UI control in picture format in the second image and the interactive target UI control in the target image. However, the first UI control in the second image is not interactive.

[0071] Alternatively, in other embodiments, although the first UI control and the target UI control display the same control information, the first UI control and the target UI control may be different types of controls; or, the first UI control and the target UI control are the same type of controls and display the same control information, but the two may differ in at least one aspect of the control border, shadow, hue, size, etc., so that users can distinguish that the control displayed on the second image is a non-interactive UI control, while the control displayed on the target image is an interactive UI control.

[0072] According to the first aspect, or any implementation of the first aspect, the size of the target UI control may be slightly larger than or equal to the size of the first UI element.

[0073] According to the first aspect, or any implementation of the first aspect above, the control information displayed by the first UI control and the target UI control may be different to some extent, for example, the card information of the weather card in the picture format is the weather information related to the shooting time and shooting location of the first image. The card information displayed by the instantiated weather card is the card information related to the current time and location of the mobile phone.

[0074] According to the first aspect, or any implementation of the first aspect above, the method further includes: storing first control metadata of the first UI control in association with the second image; and generating the target UI control based on the first control metadata.

[0075] The associative storage method may be, for example, storing the first control metadata in the metadata of the second image, or associating the first control metadata with the name of the second image.

[0076] In addition, the process of the electronic device generating the target UI control based on the first control metadata can be understood as the process of instantiating (ie, creating and displaying) the target UI control on the second image to obtain the target image.

[0077] The first control metadata may be identical to the target control metadata, i.e., the target UI control and the first UI control are the same control, or the first control metadata may be the same control type as the target control metadata, and there may be at least one difference in metadata (i.e., the properties of the control mentioned above) between the two. For example, the control sizes are different, and the size of the target UI control is larger than the size of the first UI control.

[0078] In this embodiment, the electronic device can generate a target UI control using the first control metadata associated with the second image of the first UI control in picture format to obtain a target image. In this way, the target UI control instantiated and displayed on the target image is related to the first UI control, and the instantiated interactive target UI control meets the user's demand for the UI control added to the first image.

[0079] According to the first aspect, or any implementation of the first aspect above, the method further includes: refreshing the first control metadata as the target control metadata of the target UI control.

[0080] For example, if there are certain differences between the target control metadata and the first control metadata, in order to ensure that when the second image is shared to other electronic devices, the interactive UI controls displayed on the first image by other electronic devices using the method of the present application are the same as the target UI controls, the electronic devices may refresh or replace the locally stored first control metadata with the target control metadata.

[0081] According to the first aspect, or any implementation of the first aspect above, the method further includes: in response to a received display operation on the second image, displaying a target UI control related to the first UI control on the second image to display the target image.

[0082] In this embodiment, after the user adds a first UI control in a picture format to the first image through a first user operation to obtain a second image, the electronic device can save the second image. Then when the user operates to request to display the second image, the electronic device can respond to the operation to instantiate the target UI control on the second image to display the target image. In this way, when the user needs to open a second image showing a first UI element, the electronic device can, for example, instantiate the target UI control (for example, the target UI control is the same as the first UI control) at the display position of the first UI element in the second image to display the target image. In this way, although the electronic device saves a second image with a non-interactive UI control (i.e., the first UI element), when the second image needs to be displayed, the electronic device can display an interactive target UI control on the second image to facilitate the user to interact with the target UI control in the target image.

[0083] According to the first aspect, or any implementation of the first aspect above, the method further includes: in response to a received sharing operation on the second image, sending the second image and the first control metadata associated with the second image.

[0084] In this embodiment, when the electronic device needs to share the second image with other electronic devices, the first control metadata associated with the second image can be synchronously shared with the other electronic devices. For example, if the first control metadata has been refreshed as the target control metadata in the above implementation, the electronic device can send the target control metadata to the other electronic devices. Then, after the other electronic device receives the second image and its first control metadata, when the other electronic device displays the second image, it can instantiate the target UI control according to the first control metadata to display the above target image on the other electronic device, so as to realize the sharing of pictures with interactive UI controls.

[0085] According to the first aspect, or any implementation of the first aspect above, displaying the target image includes: generating a target application; and displaying the target image in response to a received startup operation for the target application.

[0086] In an embodiment, there is no need to first add a first UI element in a picture format to the first image. Instead, when a user requests to add a UI control to the first image through a first user operation, a target UI control can be instantiated on the first image to generate a target application, which may include a first image instantiated with a target UI control, i.e., a target image. Then, each time the user starts the target application, the electronic device can display the target image stored in the target application. Compared to the above-mentioned implementation method of generating a second image and then obtaining the target image based on the second image, the method of this embodiment is more efficient in displaying an image with a UI control.

[0087] In a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes: a first display module, configured to display a target image in response to a first user operation received on a first image, wherein the target image is a first image displaying a target user interface UI control; and an output module, configured to output an interaction result related to the target UI control in response to a second user operation received on the target UI control.

[0088] According to the second aspect, the target UI control is a music control, and the music control is related to a music application running in the background when the first user operation is received.

[0089] According to the second aspect, or any implementation of the second aspect, the target UI control is related to the first image.

[0090] According to the second aspect, or any implementation of the second aspect, the control type of the target UI control is related to the image content of the first image.

[0091] According to the second aspect, or any implementation of the second aspect, the control information displayed by the target UI control is related to the image content of the first image.

[0092] According to the second aspect, or any implementation of the second aspect above, the perspective parameter of the target UI control is the same as the perspective parameter of the target image element in the first image corresponding to the display position of the target UI control.

[0093] According to the second aspect, or any implementation method of the second aspect above, the first display module is specifically used to: display a second image, wherein the second image is the first image including a first UI element, wherein the first UI element is a first UI control in a picture format; display a target UI control related to the first UI control on the second image to display a target image, wherein the first UI element is covered by the target UI control.

[0094] According to the second aspect, or any implementation of the second aspect, the first UI control and the target UI control each display the same control information.

[0095] According to the second aspect, or any implementation of the second aspect above, the electronic device also includes: a storage module for storing the first control metadata of the first UI control in association with the second image; and a generation module for generating the target UI control based on the first control metadata.

[0096] According to the second aspect, or any implementation of the second aspect above, the electronic device further includes: a refresh module, configured to refresh the first control metadata to the target control metadata of the target UI control.

[0097] According to the second aspect, or any implementation of the second aspect above, the electronic device also includes: a second display module, used to display a target UI control related to the first UI control on the second image in response to a received display operation on the second image, so as to display the target image.

[0098] According to the second aspect, or any implementation of the second aspect above, the electronic device also includes: a sending module, used to send the second image and the first control metadata associated with the second image in response to a received sharing operation on the second image.

[0099] According to the second aspect, or any implementation of the second aspect above, the first display module is specifically used to: generate a target application; and display a target image in response to a received startup operation on the target application.

[0100] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0101] In a third aspect, an embodiment of the present application provides an electronic device. The electronic device includes: a memory and a processor, the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the method in the first aspect and any one of the implementations of the first aspect.

[0102] The technical effects corresponding to the third aspect can be found in the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.

[0103] In a fourth aspect, an embodiment of the present application provides a computer-readable medium for storing a computer program, which, when executed on an electronic device, enables the electronic device to execute the method in the first aspect and any one of the implementations of the first aspect.

[0104] The technical effects corresponding to the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.

[0105] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer is caused to execute the method in the first aspect and any one of the implementations of the first aspect.

[0106] The technical effects corresponding to the fifth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.

[0107] In a sixth aspect, an embodiment of the present application provides a chip. The chip includes one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of a first electronic device and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the first electronic device executes the method in the first aspect and any one of the implementations of the first aspect.

[0108] The technical effects corresponding to the sixth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0110] Figure 1 One of the structural schematic diagrams of an electronic device is shown as an example;

[0111] Figure 2 A schematic diagram of the software structure of an electronic device is shown as an example;

[0112] Figure 3a One of the schematic diagrams of the processing process is shown as an example;

[0113] Figure 3b One of the schematic diagrams of the processing process is shown as an example;

[0114] Figure 3c One of the schematic diagrams showing an exemplary sharing process;

[0115] Figure 4 One of the schematic diagrams of the processing process is shown as an example;

[0116] Figure 5a One of the schematic diagrams of an exemplary application scenario;

[0117] Figure 5b One of the schematic diagrams of an exemplary application scenario;

[0118] Figure 5c One of the schematic diagrams of an exemplary application scenario;

[0119] Figure 5d One of the schematic diagrams of an exemplary application scenario;

[0120] Figure 6a One of the schematic diagrams of an exemplary application scenario;

[0121] Figure 6b One of the schematic diagrams of an exemplary application scenario;

[0122] Figure 6c One of the schematic diagrams of an exemplary application scenario;

[0123] Figure 6d One of the schematic diagrams of an exemplary application scenario;

[0124] Figure 7a One of the schematic diagrams of an exemplary application scenario;

[0125] Figure 7b One of the schematic diagrams of an exemplary application scenario;

[0126] Figure 7c One of the schematic diagrams of an exemplary application scenario;

[0127] Figure 7d One of the schematic diagrams of an exemplary application scenario;

[0128] Figure 8a One of the schematic diagrams of an exemplary application scenario;

[0129] Figure 8b One of the schematic diagrams of an exemplary application scenario;

[0130] Fig. 9 One of the schematic diagrams of an exemplary application scenario;

[0131] Fig.10a One of the schematic diagrams of an exemplary application scenario;

[0132] Fig.10b One of the schematic diagrams of an exemplary application scenario;

[0133] Fig.11 A schematic diagram of the structure of the device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0134] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0135] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0136] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0137] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0138] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0139] Figure 1 1 shows a schematic diagram of the structure of the electronic device 100. It should be understood that Figure 1The electronic device 100 shown is only an example of an electronic device. Optionally, the electronic device 100 may be a terminal, which may also be referred to as a terminal device. The terminal may be a cellular phone, a tablet computer (pad), a wearable device or an IoT device, a vehicle-mounted device, or other device with a display screen or an external display screen, which is not limited in this application. It should be noted that the electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0140] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0141] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0142] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0143] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0144] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0145] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0146] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0147] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0148] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0149] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0150] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0151] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0152] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0153] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0154] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0155] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0156] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0157] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0158] The camera 193 is used to capture still images or videos. The object is projected onto the photosensitive element by generating an optical image through the lens.

[0159] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0160] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 enables the electronic device 100 to execute the method in the embodiment of the present application by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0161] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0162] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0163] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0164] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0165] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0166] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0167] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0168] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0169] The layered architecture of the electronic device 100 divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.

[0170] The application layer can include a series of application packages.

[0171] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0172] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0173] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, a user interface (UI) framework module, and the like.

[0174] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0175] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0176] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0177] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0178] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0179] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0180] The gallery application may call the UI framework module to convert the UI control into a UI control in a picture format, and add the UI control in the picture format to the picture in the selected gallery application.

[0181] In addition, the gallery application can also call the UI framework module to add the location of the UI control in the picture format to the above picture when displaying the picture, and instantiate the corresponding UI control so that the UI control displayed on the picture is an interactive UI control rather than a static image. The specific processing process will be described in detail in the following embodiments.

[0182] Among them, the UI control can be any interactive UI control, such as a service card, a notification message, a live window, a floating control (such as a floating window, a floating ball, etc.), an interactive button (such as a button for calling a contact), a text box (such as a search box, etc.), an application interface (such as a chat interface, etc., without limitation).

[0183] Taking the UI control as a service card as an example, the instantiation process of the UI control is explained. Instantiating a service card at a certain location means creating a service card at the location so that the card displayed at the location is interactive.

[0184] The system library and runtime layer include the system library and Android Runtime. The system library can include multiple functional modules. For example: surface manager, media library, 3D graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc. The 3D graphics library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing, etc. The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core library contains two parts: one part is the function functions that need to be called by the Java language, and the other part is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.

[0185] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0186] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0187] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0188] A 2D graphics engine is a drawing engine for 2D drawings.

[0189] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0190] Understandably, Figure 2 The components included in the system framework layer, system library and runtime layer shown do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine some components, or split some components, or arrange the components differently.

[0191] Currently, with the development of technology, electronic devices can provide service cards to facilitate users to use corresponding functions or services of applications or electronic device systems.

[0192] Taking a mobile phone as an example, in some scenarios, a user desires to add a service card provided by the mobile phone to a picture for display. In the prior art, the user can display a screen with a service card in the picture by overlaying, cropping, and erasing a screenshot of the service card and the picture to be added with the card.

[0193] However, in the prior art, the pictures with service cards added thereto are only static images of the service cards, and are not interactive service cards.

[0194] To this end, the present application provides an image processing method and an electronic device, which can add UI controls to a picture and process the UI controls in the picture into interactive UI controls.

[0195] Among them, the UI control can be any interactive UI control, such as a service card (also called a widget in some scenarios), a notification message, a live window, a floating control (such as a floating window, a floating ball, etc.), an interactive button (such as a button for calling a contact), a text box (such as a search box), an application interface (such as a chat interface, etc., without limitation), etc.

[0196] Taking the UI control as a service card as an example, in an example 1, the electronic device can Figure 2 The gallery application shown calls the UI framework module to execute the image processing method, thereby embedding the service card into the selected picture in the form of an image, and when displaying the picture, the location of the service card can be displayed in the selected picture to instantiate the service card so that the service card displayed in the picture can be interactive.

[0197] It should be understood that the above embodiment uses the gallery application as an example to illustrate the method of the present application, but the application implemented by the method of the present application is not limited to the gallery application, but can also be other applications that can display pictures. The method is similar and will not be repeated here.

[0198] Taking the UI control as a service card as an example, in an example 2, the electronic device can instantiate a service card on the selected picture to generate a target application, which can be added to Figure 2 The target application may include a target image with an interactive service card, and the electronic device may start the target application to display the target image with the interactive service card and implement the interactive process of the service card.

[0199] The following takes the electronic device as a mobile phone as an example, combined with application scenarios 1 to 4, to illustrate the implementation process of the method in the above example 1 of the present application. When the electronic device is another type of electronic device, or when the UI control is another type of interactive UI control, the principle of the implementation process of the method of the present application is the same or similar, and will not be repeated here.

[0200] Example 1

[0201] Please refer to Figure 3a to Figure 3c , taking the mobile phone's gallery application to call the UI framework module as an example to illustrate the implementation process of the method of the present application.

[0202] in, Figure 3a It mainly describes the process of embedding the service card into the selected picture in the form of an image by calling the UI framework module in the mobile phone gallery application.

[0203] Figure 3b It mainly describes the process of displaying the position of the service card in the selected picture by the mobile phone and instantiating the service card.

[0204] Figure 3c It mainly describes the process of sharing pictures with images embedded with service cards by mobile phones.

[0205] Figure 5a to Figure 5d as well as Figures 6a to 6d To achieve Figure 3a to Figure 3c Schematic diagram of application scenario 1 of the method shown.

[0206] Please refer to the following Figure 3a to Figure 3c , Figure 5a to Figure 5d as well as Figures 6a to 6d , to introduce the method of this application.

[0207] Please refer to Figure 3a , the process may include the following steps:

[0208] S100a: The mobile phone determines a first image.

[0209] The first image is an image of the service card to be added, and the first image may be the selected picture mentioned in the above embodiment.

[0210] In some embodiments, the first image can be a current image taken by the user using a camera application on a mobile phone, or an image selected by the user from a local picture (such as a gallery application), or a screenshot image or an image shared by other users, or a frame of an image in a video, etc. There is no restriction on the source of the first image.

[0211] In addition, the present application does not impose any restrictions on the display position of the first image. For example, the first image may also be an image dragged by the user to the transfer station.

[0212] S100b, the mobile phone determines the first service card.

[0213] The first service card may be a service card created through user operation, or a service card selected by the user from among the created service cards, or a service card recommended by the mobile phone according to the first image and / or based on the real-time scenario.

[0214] For example, if the first image includes a certain building, the mobile phone may recommend a map card that matches the location of the building, so as to display the address information of the building in the map card.

[0215] For another example, if the first image includes a projection screen, the mobile phone may recommend a video card that matches the image displayed on the projection screen. For example, the card information displayed by the video card may include a video image that matches the image displayed on the projection screen in the first image.

[0216] For another example, when the mobile phone captures the first image, the real-time scene of the mobile phone is that the mobile phone is playing music. The mobile phone can recommend a music card based on the real-time scene (playing music).

[0217] In some embodiments, the card content of the first service card may be related to the first image.

[0218] For example, the card information may be related to the metadata of the first image (such as at least one item of the shooting location, shooting time, etc.). In the above example of the recommended map card, the card information displayed by the recommended map card may be the map information of the shooting location of the first image.

[0219] For another example, the card information may be related to the image frame of the first image. For example, if the first image includes a picture of a person walking, the first service card determined by the mobile phone may be a sports card, and the card information displayed by the sports card may include information on the number of steps related to the picture.

[0220] In this way, the card information and card type of the service card embedded in the image can match the scene or semantics expressed by the image.

[0221] Among them, the present application does not limit the execution order of S100a and S100b. The two can be executed in parallel or serially. The specific execution order may vary based on different application scenarios and is not limited here.

[0222] Application Scenario 1

[0223] The following uses application scenario 1 as an example to illustrate Figure 3a The implementation process of the above S100a and S100b is shown.

[0224] In application scenario 1, please refer to Figure 5a ,like Figure 5a As shown in (1), the display interface 201 of the mobile phone is the application interface of the gallery application. The display interface 201 may include multiple controls, which may include but are not limited to: picture 202, control 203, control 207, etc.

[0225] The picture 202 is a picture stored in the gallery application; the control 207 is used to share the picture 202 .

[0226] like Figure 5a As shown in (1), the user clicks the control 203 in the display interface 201, and the mobile phone can respond to the user operation and switch the display interface to Figure 5a (2) The display interface 201 shown, specifically, the mobile phone can respond to the user operation and display a control 204 in the display interface 201. The control 204 may include a rename option, an add card option 2041, a rotation option, etc.

[0227] The rename option is used to rename the name of the picture 202 , and the rotate option is used to rotate the picture 202 .

[0228] like Figure 5a As shown in (2), the user clicks control 2041, and the mobile phone can respond to the user operation and switch the display interface to Figure 5a (3) The display interface 300 shown. Figure 5a As shown in (3), the display interface 300 may include one or more controls, which may include but are not limited to: a search control 301, a map card 302, a weather card 303, a control 304 for setting a combination card, a control 305 for setting a variety of cards, a control 306 for adding a music card of a music application, a control 307 for adding a video card of a Huawei video application, a control 308 for adding a sports card of a sports health application, etc.

[0229] The search control 301 is used to search for service cards that can be provided by the mobile phone.

[0230] In addition, if Figure 5a The display interface 300 shown in (3) also includes the text "Map (9)" located below the map card 302, which is used to indicate that the mobile phone provides 9 types of map cards related to the map application, and at least one of the display style, display content, etc. of the various map cards may be different.

[0231] And, if Figure 5aThe display interface 300 shown in (3) also includes the text "Weather (3)" located below the weather card 303, which is used to indicate that the mobile phone provides three weather cards related to the weather application, and at least one of the display style, display content, etc. of the various weather cards may be different.

[0232] Users expect Figure 5a (2) Add a weather card to the picture 202 shown in FIG. Figure 5a As shown in (3), the user clicks on the weather card 303, and the mobile phone can respond to the user operation and switch the display interface of the mobile phone to the following Figure 5a (4) The display interface 401 shown in FIG. 401 may include three types of weather cards for preview, namely weather card 303, weather card 402, and weather card 403. The three types of weather cards are Figure 5a The text "Weather (3)" shown in (3) indicates three weather cards related to the weather application. And the weather card 303 displayed on the top layer among the three weather cards previewed by the mobile phone indicates the currently selected weather card.

[0233] In addition, if Figure 5a As shown in (4), the display interface 401 also displays the text "Real-time Weather" to indicate that the card information in the weather card 303, weather card 402, and weather card 403 may include: weather information provided by the weather application that matches the mobile phone system time and the mobile phone location.

[0234] Take weather card 303 as an example. The mobile phone system time is 8:00 on July 1, 2023, and the mobile phone is located in Beijing. Then the temperature information (current temperature, maximum temperature, minimum temperature, etc.) and weather reminder "lightning and rainstorm warning" provided by weather card 303 are the current temperature (27°C), maximum temperature (here 30°C) and minimum temperature (here 23°C) forecasted by the weather application for Beijing on July 1, 2023. The same is true for weather cards 402 and 403, which will not be repeated here.

[0235] In addition, if Figure 5a As shown in (4), the display interface 401 may also include a control 404, a control 405, and a control 406.

[0236] Among them, the control 405 can be used to add the selected weather card 303 to the desktop interface display.

[0237] For example, in Figure 5a In the embodiment, the shooting location of the picture 202 to be added to the service card is "Hebei Province", so in order to add the weather card to the picture 202, the card information displayed is matched with the weather of the shooting location, such as Figure 5a(4), the user can click on the control 404, and the mobile phone can respond to the user operation and refresh the display interface of the mobile phone from the display interface 401 shown in FIG. 5 (4) to Figure 5a (5) Display interface 401 shown.

[0238] For example, the phone can respond to the user clicking Figure 5a (4) The operation of the control 404 shown in FIG. Figure 5a (1) The metadata information of the picture 202 shown in FIG. 1 may include the shooting time (here, Figure 5a (1) as shown in FIG. 5( 5:10 p.m. on June 5, 2020) and the shooting location; then, the mobile phone can call the weather application to search for historical weather information with the shooting time and the shooting location; then, the mobile phone can use the historical weather information to refresh the card information shown in the weather card 303 and the weather card 402 as shown in FIG. 5( 4) to obtain the following information: Figure 5a (5) shows the card information of weather card 303 and weather card 402. Figure 5a The weather card 303 and the weather card 402 shown in (5) each display the weather information of Hebei Province on June 5, 2020.

[0239] In this embodiment, due to Figure 5a (4) The weather card 403 shown shows the weather forecast information for Beijing for today, tomorrow and the day after tomorrow, that is, it is used to represent the weather forecast information for multiple days. Therefore, the weather card 403 is not suitable for displaying the weather information of the picture 202. Figure 5a The weather card shown in (5) no longer includes weather card 403.

[0240] In addition, compared with Figure 5a (4) Figure 5a As shown in (5), the text "Real-time Weather" displayed on the display interface 401 is also refreshed to "Photo Weather" to indicate that Figure 5a (5) The card information in the displayed weather card is all weather information about the photo, here it is weather information related to the picture 202.

[0241] In addition, compared with Figure 5a (4) Figure 5a As shown in (5), the display text of control 404 switches from "matching photos" to "real-time weather". Figure 5a (5) When the control 404 shown in FIG. 5 is displayed, the mobile phone can restore the display interface to the following state: Figure 5a The display interface shown in (4) displays the real-time weather information of the mobile phone's location in the previewed three weather cards.

[0242] Then, if Figure 5a As shown in (5), the user clicks on control 406, and the mobile phone can respond to the user operation and switch the display interface of the mobile phone to Figure 5a (6) The display interface 201 shown.

[0243] Specifically, the mobile phone can respond to the user clicking the control 406 to execute the following operations: Figure 3a The following S101a, S101b, S101c, S102, S103, S104 are shown, so that Figure 5a (5) The weather card 303 shown is converted to Figure 5a (6) and insert the image 2051 between the foreground image 202a and the background image 202b of the image 202 to obtain the image 206.

[0244] Combine the following Figure 5a , Figures 6a to 6c , come on Figure 3a S101a, S101b, S101c, S102, S103, and S104 shown are introduced.

[0245] like Figure 3a As shown, after S100a, the method further includes S101a.

[0246] S101a, the mobile phone may divide the first image into a foreground image including foreground elements and a background image including background elements.

[0247] Combined with Figure 5a , please refer to Figure 6a , Figure 6a Shows the Figure 5a The process of performing the segmentation operation S101a on the picture 202 in order to obtain a foreground image including foreground elements and a background image including background elements.

[0248] In S101a, the phone can Figure 6a The picture 202 shown in (1) (here an example of the first image) is subjected to semantic segmentation and depth analysis to obtain multiple layers, and the elements included in each of the multiple layers are different. For example, the multiple layers are respectively a layer including human elements, a layer including sky elements (here blue sky and white clouds), and a layer including mountain elements. The areas of each layer except the included elements are transparent areas, and the size of each layer is the same as the size of the picture 202.

[0249] In S101a, the mobile phone can determine a background layer from the multiple layers based on the elements in each layer and / or the semantics of the layers obtained above, so as to obtain a background image including background elements. For example, when determining the background layer according to the elements in the layers, the mobile phone can determine a layer with relatively simple elements and fewer elements in the layers as the background layer, such as a relatively pure color block.

[0250] For example, in Figure 6a In the scene shown, the mobile phone can combine the semantics of the layers (such as sky, ground, wall, water surface, etc.) to determine that the layer including the sky element and the layer including the mountain element are both background layers; then, the mobile phone can synthesize the two background layers to obtain the following Figure 6a (3) shows a background image 202b including background elements. The background image 202b includes background elements (blue sky, white clouds and mountain elements) 2022. The area other than the background elements 2022 in the background image 202b (eg, the area where the foreground element 2021 is located) is a transparent area.

[0251] In S101a, the mobile phone may determine the layers except the background layer among the obtained multiple layers as foreground layers, or determine the foreground layers in combination with the semantics of the layers to obtain a foreground image including foreground elements.

[0252] For example, in Figure 6a In the scene shown, the mobile phone can combine the semantics of the layers to determine the layer including the human element as the foreground layer, and determine the foreground layer as the foreground image 202a including the foreground element 2021. The foreground image 202a includes the foreground element (human element) 2021, and the area in the foreground image 202a except the foreground element 2021 (for example, the area where the background element 2022 is located) is all transparent area.

[0253] In this way, the mobile phone can call the UI framework module through the gallery application to divide the picture 202 into a foreground image 202a including foreground elements and a background image 202b including background elements.

[0254] It should be understood that Figure 5a In the application scenario shown, the mobile phone executes Figure 3a The timing of segmenting the first image in S101a is not limited to responding to a user clicking Figure 5a The operation of the control 406 shown in (5) can also be performed in response to the user clicking Figure 5a (2) The control 2041 shown in FIG. 2 is operated to execute S101a. In other words, once the user selects the first image to be added, the mobile phone can segment the first image to obtain the corresponding foreground image and background image.

[0255] It should be understood that although in the application scenario related to the above example 1, the first image (eg Figure 5a The picture 202 shown in the figure is only divided into a foreground image and a background image. However, in some embodiments, the foreground element may correspond to multiple image objects, so the foreground images obtained by segmentation may also be multiple, for example, each foreground image includes one foreground element, and the other areas are transparent. Similarly, the first image may also be divided into multiple background images, for example, each background image includes one background element, and the other areas are transparent. Different background images may involve different background elements. For example, in Figure 5a , Figure 6b In the application scenario, the mobile phone can split the image including the sky element into a background image, and the other areas are transparent; and the mobile phone can split the image including the mountain element into a background image, and the other areas are transparent; if there are multiple people in the photo, at least one person can constitute a foreground image, and the area in the foreground image except the at least one person is a transparent area.

[0256] It should be understood that the above embodiment uses a person object as a foreground element as an example. In other embodiments, the foreground element can be any type of image object such as an animal, a plant, an object, a building, etc. The present application does not limit the type of image object involved in the foreground element. In addition, the foreground element in the first image is not limited to the single person exemplified above. The image object used as the foreground element can be multiple, such as multiple people, or people and animals, etc., which is also not limited here.

[0257] In addition, the foreground image and the background image obtained by the mobile phone after segmenting the first image have the same format as that of the first image.

[0258] After S100b, the mobile phone may further execute S101b.

[0259] S101b, the mobile phone may convert the first service card into a second image.

[0260] Among them, the first service card is a control rather than a picture. In order to embed the first service card into the first image, the gallery application of the mobile phone can call the UI framework module to convert the first service card into a picture format to obtain a second image. The second image has the same format as the above-mentioned first image. In this way, images with the same format are easy to synthesize through the following S102.

[0261] For example, if the first image is in Scalable Vector Graphics (SVG) format, the mobile phone can convert the first service card into SVG format; if the first image is in JPG format, the mobile phone can convert the first service card into JPG format.

[0262] The method of converting the service card into an image may differ based on the format of the first image. For example, if the image format of the first image is a bitmap image, the mobile phone may rasterize the first service card to convert the first service card represented in a vector graphics format into a bitmap.

[0263] Please refer to Figure 6b , the mobile phone can use the above S101b to Figure 5a The weather card 303 shown in (5) is converted into a picture to obtain image 205. After the weather card 303 is converted into image 205, the card information displayed in image 205 is the same as the card information displayed in the weather card 303. The difference is that image 205 is no longer a service card and cannot be interacted with.

[0264] Optionally, after S101b, the process may further include S101c.

[0265] S101c, the mobile phone may adjust the second image to a third image.

[0266] In some embodiments, the mobile phone can determine a first area of ​​the image in the first image where the service card is inserted; and the mobile phone can refer to the image parameters of the first image related to the first area to adjust the corresponding image parameters of the service card in picture format (the second image here), so that after the second image with adjusted image parameters (named as the third image here) is inserted into the first image, the third image can be more integrated with the picture of the first image, so that the service card inserted into the first image can be displayed without being abrupt on the entire image.

[0267] In some embodiments, as described above, the first image of the service card to be added can be divided into a foreground image and a background image. Then, when the mobile phone determines the first area of ​​the image to which the service card is to be inserted, the service card in the corresponding picture format can be inserted between the foreground image and the background image, and the foreground element in the foreground image can block the area ratio of the service card within a preset threshold range, for example, 3% to 20% of the area of ​​the service card is blocked by the foreground element, so as to determine the position of the first area. The preset threshold range can be flexibly changed according to actual needs and is not limited here.

[0268] In some embodiments, the mobile phone may also determine the first area based on the position or area selected by the user for the first image. For example, if the user selects a coordinate position of a service card to be added in the first image, the mobile phone may determine the first area including the coordinate position based on the coordinate position.

[0269] In some embodiments, the image parameters of the first image related to the first area may include but are not limited to: the size of the first area, the shape of the first area, the perspective parameters of the background elements and / or foreground elements located in the first area (which may include at least one of the perspective direction and the perspective angle), the color, hue, transparency, lighting characteristics and other information of the foreground elements and / or background elements related to the first area.

[0270] For example, a solution for adjusting the second image by referring to the perspective parameter (which may include at least one of the perspective direction and the perspective angle) of the background element located in the first area by the mobile phone will be described below. Figures 7a to 7d The application scenario 2 is described in detail and will not be repeated here.

[0271] The scheme of adjusting the second image by referring to the perspective parameters of the foreground element located in the first area by the mobile phone is the same as the principle of adjusting the second image by referring to the perspective parameters of the background element mentioned above. For example, the position of the service card to be added in the first image (such as the first area mentioned above) is not limited to being located between the background image where the background element is located and the foreground image where the foreground element is located, but can also be located in the upper layer (such as the top layer) of the foreground image where the foreground element is located, so that the second image can be adjusted by referring to the perspective parameters of the foreground element and / or the background element located in the first area.

[0272] The lighting characteristics may be at least one lighting characteristic such as direct lighting, reflected lighting, shadow, etc.

[0273] For example, the background image of the first image includes a sun element, and the light emitted by the sun element shines on the human element in the foreground image, and the insertion position of the second image converted from the first service card in the first image (such as the first area mentioned above) is on the illumination path. Then the mobile phone can use the Ray Tracing (RT) technology to calculate the direct illumination characteristics of the area or position on the illumination path in the second image, and render the direct illumination characteristics of the position on the illumination path in the second image to obtain a third image, so that after the third image is inserted into the first area in the first image, the position on the illumination path in the third image can display a direct illumination visual effect. Optionally, since the calculation of illumination characteristics requires a high computing power, the mobile phone can interact with the server to realize the calculation of the direct illumination characteristics of the image, so as to obtain the rendering result of the third image, and then use the rendering result to obtain the third image.

[0274] Of course, the way in which a mobile phone renders a third image with a visual effect having corresponding lighting characteristics on a service card in picture format is not limited to the above example. Other technologies such as AIGC can also be used to calculate the lighting characteristics of the service card (or its converted picture).

[0275] Please refer to Figure 6c , Figure 6c (1) and Figure 6c (3) shows the Figure 6a The foreground image 202a and the background image 202b obtained by the segmentation process shown in FIG. Figure 6c (1) and Figure 6c As shown in (3), the first area to be inserted into the service card is shown in the foreground image 202a and the background image 202b with a selection box. The size of the first area is Figure 6b The service cards in picture format shown, ie, images 205, have the same size.

[0276] In this application scenario 1, if Figure 6c As shown in (3), the background elements in the first area are the sky and the mountain elements, and there is no tilt angle. Therefore, in this application scenario 1, the mobile phone does not need to Figure 6b The image 205 shown has the perspective parameters adjusted.

[0277] In this application scenario 1, please refer to Figure 6c (1) and Figure 6c (3), and Figure 6b The mobile phone may refer to the hue (optionally including color, transparency, etc.) of the background elements located in and around the first area, and the hue of the foreground elements located in and around the first area, to adjust the hue of the image 205 through S103c to obtain Figure 6b and Figure 6c (2) The image 2051 shown. The color tone of the image 2051 can match the color tones of the foreground elements and background elements in and around the first area.

[0278] S102, the mobile phone synthesizes the foreground image including the foreground elements, the background image including the background elements and the third image (when S101c is not included, it is synthesized with the second image) into a target image, so that the above-mentioned first image can be refreshed as the target image.

[0279] Among them, the synthesized target image may include three layers from top to bottom, namely, an upper layer (also referred to as the first layer hereinafter), a middle layer (also referred to as the second layer hereinafter) and a lower layer (also referred to as the third layer hereinafter).

[0280] Among them, the foreground image is the upper layer, the third image (or the second image) is the middle layer, and the background image is the lower layer.

[0281] In application scenario 1, please continue to refer to Figure 6c When the mobile phone combines the third image after the tone adjustment into the first image, the Figure 6a The foreground image 202a obtained by the process of segmentation shown is Figure 6c The first layer shown in (1) uses the background image 202b as Figure 6c The third layer shown in (3) uses the image 2051 after the tone adjustment as Figure 6c (2) The second layer shown in FIG. 1 is synthesized by S102 in the order of layers from top to bottom to obtain the following: Figure 6c (4) An image 206 (an example of a target image) is shown.

[0282] like Figure 6c As shown in (4), the lower right corner of the image 2051 inserted in the image 206 is blocked by the foreground element (here, the head of the person) in the foreground image 202a, and the image 2051 can block the background element located in the first area of ​​the background image 202b.

[0283] In other embodiments, the mobile phone may not split the first image, and may directly insert a service card in picture format, such as the second image or the third image, into the corresponding position of the first image, so that the image of the service card may be located in the upper layer of the first image.

[0284] The mobile phone refers to the image parameter (here, the color tone) of at least one of the foreground element and the background element related to the first area of ​​the first image to Figure 6b The service card converted into a picture format as shown (here is the second image) is toned so that the toned-adjusted image 2051 is embedded into the first image, so that the image of the service card embedded in the first image can be integrated and matched with the tones of the surrounding image elements.

[0285] Back to Figure 5a , the phone can respond to the user clicking such as Figure 5a (5) The operation of the control 406 shown in FIG. Figure 6c The image 206 shown in (4) is displayed as Figure 5a (6) The display interface 201 shown.

[0286] In some embodiments, Figure 5a (6) The image 206 in the display interface 201 shown may be a Figure 5a(1) is the replacement result of the image 202. In some embodiments, the image 206 may also be a picture saved in the gallery application.

[0287] Optionally, after S102, the process may further include S103.

[0288] S103, the mobile phone may determine card metadata related to the target image based on the target image, the third image and the first service card.

[0289] The card metadata may include but is not limited to at least one of the following: card type, card information, card size (such as the size of the first area mentioned above), card position (such as the position of the first area in the first image), card shape (such as the shape of the first area mentioned above), and card image parameters.

[0290] Optionally, the card metadata may also include segmentation data related to the above S101a, and the segmentation data is used to indicate how to segment the first image to obtain a foreground image and a background image. Figure 3b In the process of instantiating the second service card for the target image, the mobile phone can refer to the segmentation data to add the instantiated second service card between the foreground image and the background image in the target image. Figure 3c In the process of sharing the target image shown, the mobile phone 2 does not need to configure a segmentation algorithm to segment the target image, but only needs to use the segmentation data in the card metadata to obtain the foreground image and the background image to instantiate the service card.

[0291] The image parameters of the card are the image parameters of the third image after the service card in picture format is adjusted to the third image through the above S101c, or the image parameters of the card may also be the image adjustment parameters (such as size adjustment parameters, transparency adjustment parameters, etc.) used when the service card in picture format is adjusted to the third image through the above S101c, wherein the image adjustment parameters may indicate information such as the adjustment direction and adjustment amount of the image parameters, and the image adjustment parameters are relative to the above image parameters, and the image adjustment parameters are absolute image parameters.

[0292] In some embodiments, the image parameters may include, but are not limited to, information such as color, hue, transparency, lighting characteristics, and perspective parameters.

[0293] For example, the mobile phone can determine the card type and card information of the first service card based on the first service card determined in S100b, and the mobile phone can determine the card perspective parameters, card size, card position, card color, card tone, card lighting characteristics, card transparency, etc. in the card metadata based on the perspective parameters of the third image used when performing the synthesis operation of S102, the size of the third image, and the synthesis position (the first area mentioned above) in the first image, color, hue, lighting characteristics, transparency, etc.

[0294] For example, Figure 5a and Figures 6a to 6c In the application scenario 1 shown, the card metadata related to the image 206 determined by the mobile phone includes: the card type is a weather card, the card information is Figure 6c (4) The image 2051 shown in FIG. 1 shows weather information, the card size is the size information of the image 2051, and the card position is the display position of the image 2051 (here, between the foreground image 202a and the background image 202b). Figure 6c The information of the first area indicated by the dashed box shown in the figure), the image parameters of the card (here, the color tone parameters, such as Figure 6b After the hue of image 205 is adjusted, the hue parameters of image 2051).

[0295] Optionally, after S103, the method may further include S104.

[0296] S104: The mobile phone may write the card metadata into the metadata of the target image.

[0297] The mobile phone can write the card metadata obtained in S103 into Figure 5a In the metadata of the image 206 shown in (6), the metadata of the image 206 embedded with the service card in the form of an image can carry the metadata information of the service card indicated by the target image in which it is embedded.

[0298] Alternatively, in other embodiments, the mobile phone may also write the obtained card metadata into a file with the same name as the image 206 , so that the card metadata can be stored in association with the image 206 .

[0299] It should be understood that Figure 5a , Figures 6a to 6c The embodiment is illustrated by taking the service card added to the picture as a system preset service card as an example, but the service cards involved in the various embodiments of the present application are not limited to system preset service cards, but can also be service cards generated by other known or future developed methods, which are not limited here.

[0300] For example, a mobile phone may generate a service card related to the control content of the movable control in response to a user's operation on the movable control, and the service card may be used as a service card for adding to a picture in various embodiments of the present application. The control content may include information displayed on the movable control and / or information related to the movable control but not displayed. The movable control may be a notification message, a floating control, a transfer station control, etc.

[0301] In the embodiment of the present application, in order to add a service card to the first image, the mobile phone may first convert the service card to be added into a picture format, and then convert the service card in the picture format (eg Figure 3a The second image or the third image shown in the figure is added to the first image to obtain the target image, and the metadata information of the service card is associated with the target image and saved. In this way, each time the target image needs to be displayed, the mobile phone can refer to the metadata information, create a service card with the same display effect as the second image (or the third image), and add the service card to the target image, so that the interactive service card is displayed at the position where the service card in picture format is displayed in the target image, and the instantiation of the service card in the image (for example, creating a service card) can be quickly realized, so that the instantiated service card in the target image can be interactive.

[0302] Combined with the above Figure 3a , Figure 5a , Figures 6a to 6c ,exist Figure 3a After the process, please refer to Figure 3b , which shows that the mobile phone Figure 3a The generated target image is used to instantiate the service card. Figure 5b as well as Figure 6d , come on Figure 3b The application scenario 1 of the process is introduced.

[0303] Please refer to Figure 3b , the process may include the following steps:

[0304] S300, the mobile phone determines the target image.

[0305] For example, when the phone displays Figure 5a (6) After the display interface 201 is displayed, the mobile phone switches the application running in the foreground from the gallery application to another application. After a period of time, the user opens the gallery application to display the following Figure 5b (1) The display interface 407 shown.

[0306] The display interface 407 is an application interface of the gallery application, and the display interface 407 includes reduced images of multiple images stored in the gallery application, and the reduced images of the multiple images include image 2061. Figure 5a , Figures 6a to 6c The reduced image of the image 206 stored in the gallery application by the process of FIG.

[0307] in, Figure 5b (1) The image 206 pointed to by the dotted arrow is the original image after the image 2061 is enlarged. Figure 5a (6) is the same as the image 206 shown.

[0308] Then, the user clicks Figure 5b (1) to request to display image 206 (an example of a target image), the mobile phone can determine the image 206 to be displayed in response to the user's click operation, and execute the following S301 to S303, thereby changing the display interface of the mobile phone from Figure 5b The display interface 407 shown in (1) is switched to Figure 5b The display interface 201 shown in (2) is used to display the image 206, wherein a service card is instantiated at the position (such as the first area) where the image 2051 is displayed in the displayed image 206, which is the weather card 309.

[0309] Next, we combine S301 and S303 and Figure 6d , to illustrate the phone will display the interface from Figure 5b The display interface 407 shown in (1) is switched to Figure 5b (2) The processing process of the display interface 201 shown.

[0310] S301, the mobile phone obtains card metadata related to the target image.

[0311] like Figure 3a As described in S104 shown in FIG. 1 , the card metadata is stored in the image metadata of the target image. Therefore, the mobile phone can Figure 5b The card metadata is extracted from the image metadata of the original image (i.e., image 206) corresponding to the image 2061 shown in (1).

[0312] In some embodiments, when Figure 3a In the process shown, if the card metadata related to the image 206 is not saved, or the saved card metadata is incomplete, the mobile phone can also perform image analysis on the image 206 to obtain the card metadata.

[0313] As described above, the card metadata may include but is not limited to at least one of the following: card type, card information, card size (such as the size of the first area), card position (such as the position of the first area in the first image), card shape (such as the shape of the first area), and card image parameters. In some embodiments, the image parameters may include but are not limited to: color, hue, transparency, lighting characteristics, perspective parameters and other information.

[0314] For example, a mobile phone can Figure 5b (1) The image 206 shown is analyzed to determine that the image 2051 with a rectangular frame shape is an image of a service card; and the mobile phone determines that the image 2051 is an image of a weather card by semantically analyzing the image elements in the image 2051, such as the sun element, the temperature element, and the place name element, so as to determine that the image 2051 is an image of a weather card, and the card type is determined to be a weather card. The mobile phone can also determine the card information, card size, card position, card shape, card image parameters, etc. by the same or similar methods.

[0315] Compared with obtaining card metadata through image sharing, reading the card metadata related to the target image is more accurate and efficient. The corresponding method can be flexibly selected to obtain card metadata based on the computing power and actual needs of the mobile phone.

[0316] S302: The mobile phone may instantiate a second service card for the target image based on the card metadata, so that the target image is displayed with an interactive second service card.

[0317] For example, Figure 6d As shown in (1), the mobile phone can create a second service card (for example, calling an interface of a weather application for creating a service card) based on the card metadata related to image 2051, which is weather card 309 here, wherein the size, shape, card information, image parameters, etc. of weather card 309 are the same as the corresponding parameters of image 2051.

[0318] The image parameters may include but are not limited to: color, hue, transparency, lighting characteristics, perspective parameters and other information.

[0319] In one example, a mobile phone may call an interface of a weather application for creating a service card, and use the card size and card information as parameters to create a weather card with the card information and a card size as the card size in the parameters; then, the mobile phone adjusts the shape and image parameters of the created weather card based on the card shape and image parameters in the card metadata to obtain the weather card 309.

[0320] In another example, the mobile phone calls an interface for creating a weather service card. For example, the interface supports inputting card shape and image parameters. The interface can carry the above card metadata, so that the above weather card 309 can be directly created.

[0321] In this embodiment, the display effect of the image 2051 is the same as the display effect of the weather card 309 instantiated by the mobile phone, but the weather card 309 is an interactive service card, and the image 2051 is only a picture format of the weather card 309.

[0322] In other embodiments, the display effect of the weather card 309 may be different from that of the image 2051. For example, the line color, thickness and other parameters of the rounded rectangular outline of the weather card 309 are different from the line color and thickness of the rounded rectangular outline of the image 2051.

[0323] In this way, the mobile phone displays Figure 5a (6) When the mobile phone sees the image 206 shown in FIG. Figure 3b The process is to instantiate the weather card 309 at the location of the image 2051 in the image 206, so as to change the display interface from Figure 5a (6) The display interface 201 shown switches to Figure 5b (2) The display interface 201 shown in FIG. 2 is used to display the weather card 309. In this way, the user can see the process of the image 2051 changing into the weather card 309 from the image 206 displayed in the display interface 201 of the mobile phone. For example, there is a difference in the color of the lines of the rounded rectangular outline between the image 2051 and the weather card 309. In addition, in some embodiments, when the mobile phone responds to the user clicking Figure 5b (1) The user operates the image 2061 shown in FIG. 1 to change the display interface of the mobile phone from Figure 5b The display interface 407 shown in (1) is switched to Figure 5b (2) In the process of displaying the interface 201 shown in FIG. Figure 5b The display interface 407 shown in (1) switches to Figure 5a (6) The display interface 201 is used to display the original image corresponding to the image 2061, that is, the image 206, and then the display interface is changed from Figure 5a (6) The display interface 201 shown switches to Figure 5b The display interface 201 shown in (2) instantiates a weather card 309 at the corresponding card position of the image 206.

[0324] Whether the display effects of the image 2051 and the weather card 309 are the same can be flexibly configured according to actual needs, and this application does not impose any restrictions.

[0325] like Figure 6dAs shown, the phone can Figure 6d The weather card 309 created as shown in (1) is inserted into the image 206 (an example of the target image) to obtain Figure 6d (2) shows an image 206 with an interactive weather card 309 (here is an example of a target image).

[0326] Specifically, Figure 6d As shown in (1), the mobile phone can insert the weather card 309 into the card according to the card position in the card metadata. Figure 6c (1) The first layer and Figure 6c (2) between the second layer shown, that is, the weather card 309 is inserted between the foreground image 202a and the image 2051 of the weather card in picture format, and the insertion position of the weather card 309 is Figure 6c The first area indicated by the dashed box shown in FIG. Figure 6d (2) A service card, specifically a weather card 309, can be instantiated at the position where the image 2051 is originally displayed in the image 206, so that the weather card 309 can cover the image 2051. Figure 5b (2) The weather card 309 (i.e. Figure 6d (2) shows the weather card 309), the mobile phone can perform interactive operations related to the service card 309 to display the interactive results related to the service card 309. For example, when the user clicks Figure 5b (2) The weather card 309 shown in FIG. 2 , the mobile phone can respond to the user operation by calling the weather application through the weather card 309 to display the application interface of the weather application, which can be an example of the above-mentioned interaction result.

[0327] For example, the application interface may include weather information related to the weather information displayed in the weather card 309, such as the weather forecast information for Hebei in the past three days; or the application interface may display more fine-grained detailed information about the weather information in the weather card 309, such as including not only the location information and temperature information displayed in the weather card 309, but also information such as the wave index and clothing recommendations related to the location; or the application interface may also display the forecast weather information related to the current location of the mobile phone. There is no limitation here.

[0328] In the above Figure 6d In the application scenario, the mobile phone can create a new second service card based on the card metadata, and insert the second service card into the corresponding position of the target image to instantiate the service card for the target image.

[0329] In other embodiments, the mobile phone can also display a service card in a picture format (eg Figure 6d The corresponding object in the image 2051) shown in the figure adds a touch event to achieve the effect of instantiating a second service card in the target image.

[0330] For example, a mobile phone can Figure 5a (6) The location information in the image 2051 shown in FIG. 6 (for example, adding a touch event to “Hebei”) can be used to call the interface of the weather application to display weather information (the location information can be passed into the interface as a parameter), thereby implementing the process of instantiating the image 2051 into a service card 309, so that the display interface of the mobile phone can be changed from Figure 5a (6) Switch to Figure 5b (2). When the user clicks Figure 5b (2) When the "Hebei" object in the weather card 309 shown in FIG. 1 is touched, the touch event can be triggered to display the information interface about the weather forecast for Hebei provided by the weather application.

[0331] It should be understood that Figure 3b The triggering time of the process shown is when the target image needs to be displayed. In other words, the instantiation process of the service card shown in FIG. 3 is valid once, that is, each time the user requests to display a target image through an operation, the mobile phone needs to execute once. Figure 3b The above process shown is to instantiate a service card for the image 206 according to the card metadata, so that a service card can be displayed between the foreground image and the background image of the image 206, and the card information, size, shape, perspective parameters, etc. of the service card are the same as the information of the service card in the image format originally displayed in the target image. When the target image instantiated with the second service card is canceled (for example, the user switches the picture displayed in the gallery application from the target image to another picture), the mobile phone can cancel the second service card instantiated on the target image and execute the mobile phone. Figure 3b The process of the process is closed to ensure that the second service card displayed on the target image is generated in real time each time the target image is displayed.

[0332] In some embodiments, Figure 5c A schematic diagram showing a sharing scenario of an image with a service card (or a service card in a picture format) is shown.

[0333] Please refer to Figure 5c , Figure 5c The display interface 201 of (1) is an image display interface of the gallery application. The display interface 201 is similar to the above-mentioned Figure 5b (2) The display interface 201 shown is the same interface. The user clicks Figure 5c (1) shown in the control 207, such as Figure 5cAs shown in (2), the mobile phone may display a window 408 on the display interface 201 in response to the user operation. The window 408 may include one or more controls, which may include but are not limited to: image 206 and thumbnails of other pictures stored in the gallery application (such as image 4091, image 4092), information application icon 410, etc.

[0334] in, Figure 5c The image 206 shown in (2) is in a selected state as indicated by an arrow, indicating that the image 206 is a picture to be shared. Figure 5c (2) The enlarged view of the image 206 is Figure 5a (6) The image 206 shown, i.e., the image 206 to be shared, does not include the instantiated Figure 5c In the service card 309 shown in (1), the shared image 206 is still the image 206 of the service card (ie, image 2051 ) with a picture format stored in the gallery application.

[0335] In other embodiments, when the user clicks Figure 5a (6) When the control 207 shown in FIG. 6 is displayed, the mobile phone will also respond to the user operation and display Figure 5c (2) The window 408 shown in FIG. Figure 5a (6) The image 206 shown is shared.

[0336] Continue to refer to Figure 5c (2) When the user clicks the information application icon 410, the mobile phone can respond to the user operation and change the display interface from Figure 5c (2) The display interface 201 shown switches to Figure 5c (3) The display interface 412 shows a list of contacts that the user has recently chatted with using the messaging application, which are contact 1 indicated by control 413 and contact 2 indicated by control 414.

[0337] Then, the user clicks on control 413, such as Figure 5c As shown in (3), the mobile phone can respond to the user operation and display a window 415 on the display interface 412. The window 415 may include but is not limited to thumbnail 4151, send control 4152 and other controls. Figure 5a (6) is a thumbnail of the image 206 shown.

[0338] Next, the user clicks on control 4152, and the mobile phone can respond to the user operation and change the display interface from Figure 5c (3) The display interface 412 shown switches to Figure 5c(4) The display interface 416 is a conversation interface between the user of the mobile phone (also referred to as mobile phone 1) and the contact 1, wherein the display interface 416 displays the image 206 shared by the user of the mobile phone 1 to the contact 1 (for example, using mobile phone 2). In order to facilitate the user to view the shared image 206 in the display interface 416, Figure 5c (4) The dotted arrow indicates the enlarged image of the shared image 206. Figure 5a The image 206 shown in (6) is the same, that is, the image 206 shared by the user to contact 1 contains a service card in a picture format, namely, image 2051 .

[0339] In addition, if Figure 3c In the mobile phone 1, in response to the user clicking Figure 5c (3) When the control 4152 shown in FIG. Figure 5a (6) The image 206) and the metadata of the target image are sent to the mobile phone 2 used by the contact 1. Since the card metadata related to the image 206 is written into the metadata of the image 206, the card metadata related to the image 2051 is also shared with the mobile phone 2.

[0340] exist Figure 5c After the process, please refer to Figure 5d , which shows a schematic diagram of the interface of the mobile phone 2 used by the above-mentioned contact 1. Figure 5d As shown in (1), mobile phone 2 displays an application interface of an information application, here display interface 416, which includes the conversation content between the user of mobile phone 2 (i.e., the above-mentioned contact 1) and contact 3, wherein the contact 3 is the above-mentioned Figure 5c The user of mobile phone 1 is shown. Figure 5d As shown in (1), the display interface 416 includes an image 206 shared by a user of mobile phone 1 (contact 3) to a user of mobile phone 2 (contact 1), and the image 206 may include a foreground image 202a, a background image 202b, and an image 2051 inserted between the foreground image 202a and the background image 202b.

[0341] Then, the user clicks Figure 5d (1) as shown in the image 206, the mobile phone 2 can respond to the user operation and execute Figure 3b The process shown in FIG. 1 is to use the card metadata in the metadata of the image 206 to instantiate the weather card 309 for the image 206, and change the display interface of the mobile phone from Figure 6d (1) The display interface 416 switches to Figure 6d(2) The display interface 417 , wherein the display interface 417 displays the image 206 instantiated with the weather card 309 .

[0342] Thus, in this embodiment, due to Figure 3a The card metadata generated in the process is written into the metadata of the target image. Then, when mobile phone 1 shares the target image with the service card to other electronic devices (such as mobile phone 2), the card metadata can be shared to other electronic devices through the metadata of the target image. In this way, mobile phone 2 can use the shared target image and its metadata to display the target image when it needs to. Figure 3b The process is used to instantiate the service card so that the target image can display an interactive service card.

[0343] In some embodiments, the card metadata is carried in the metadata of the target image, so that the data shared from mobile phone 1 to mobile phone 2 includes the target image and its metadata. In this way, the file data only has one file where the metadata of the target image is located, and fewer files are transmitted, which is conducive to data sharing.

[0344] In another embodiment, the card metadata of the target image may also be stored separately instead of being written into the metadata of the target image. In this way, when the target image is shared by a mobile phone, the shared file data includes not only the metadata of the target image, but also the card metadata associated with the target image. In this way, multiple file data are transmitted between different electronic devices.

[0345] In actual applications, the mobile phone can flexibly determine to store the card metadata separately as a file or write it into the metadata of the target image according to the needs, and there is no limitation here.

[0346] Application Scenario 2

[0347] In the above application scenario 1, Figure 5a to Figure 5d ,and Figures 6a to 6d As an example, the above-mentioned Figure 3a , Figure 3b , Figure 3c implementation process.

[0348] In this application scenario 2, Figures 7a to 7d Take the example to describe the above Figure 3a to Figure 3b implementation process.

[0349] Most of the implementation processes of this application scenario 2 are the same as those of the above application scenario 1, with the following three differences:

[0350] Difference 1, in passing Figure 3aWhen determining the first service card in S100b, the mobile phone may recommend service cards matching the first image, so that the user can select the first service card to be added to the first image from the recommended service cards.

[0351] Difference 2: The mobile phone is passing Figure 3a When the second image is adjusted in step S101c, the perspective parameters of the second image may be adjusted so that the perspective parameters of the adjusted third image match the perspective parameters of the background elements in the background image obtained by segmenting the first image.

[0352] Difference 3: The mobile phone is passing Figure 3b When the second service card is instantiated for the target image in S302 shown, the mobile phone may first create the above-mentioned first service card based on the card information and card type in the card metadata, and then adjust the first service card in combination with the perspective parameters, card size, card shape and other information in the card metadata to obtain a second service card with the same perspective parameters, size and shape as the service card in the picture format displayed in the target image, and finally, insert the second service card into the corresponding card position in the target image.

[0353] Combine the following Figure 3a to Figure 3b , come on Figures 7a to 7d The application scenario 2 shown is described, and the differences between it and the above-mentioned application scenario 1 are mainly introduced, and the similarities are not repeated.

[0354] Please refer to Figure 7a ,like Figure 7a As shown in (1), the display interface 201 of the mobile phone is the application interface of the gallery application, and the display interface 201 may include but is not limited to: image 502 and controls 203 and 207. The scene of image 502 is a person walking in a gym, and the background of the person is the curved window of the building where the gym is located.

[0355] The user clicks on the control 203, and the mobile phone can respond to the user operation by displaying the following Figure 7a (2) The display interface 201 shown in FIG. 2 may further include a control 204. When a user clicks on the add card option 2041 in the control 204, the mobile phone may determine the card in response to the user operation. Figure 7a (2) The scene of the image 502 shown matches the type of service card (here, the music card and the sports card), and based on the type of the service card, the display interface is switched to Figure 5a (3) The display interface 3010 shown is used to recommend adding music cards or sports cards to the user.

[0356] like Figure 7aAs shown in (3), the display interface 3010 may include but is not limited to a search control 301, a control 306 for adding a music card of a music application, and a control 308 for adding a sports card of a sports health application. Among them, the control 306 is a control related to the music card, and the control 308 is a control related to the sports card.

[0357] like Figure 7a As shown in (3), the user clicks control 308, and the mobile phone can respond to the user operation and switch the display interface to Figure 7a The display interface 401 shown in (4) may include two types of sports cards for preview, namely sports card 503 and sports card 504. Among the two types of weather cards previewed by the mobile phone, the sports card 503 displayed on the top layer indicates the currently selected weather card.

[0358] In addition, if Figure 7a As shown in (4), the display interface 401 also displays the text "Daily Exercise" to indicate that the card information displayed on each of the above-mentioned exercise cards 503 and 504 may include: information on the number of exercise steps of the user on that day recorded by the sports health application.

[0359] like Figure 7a As shown in (4), the user clicks on control 406, and the mobile phone can respond to the user operation to perform the following operations: Figure 3a The above-mentioned S101a, S101b, S101c, S102, S103, S104 shown in the figure are Figure 7a (4) The sports card 503 shown is converted to Figure 7a (5) and insert the image 5051 into Figure 7a (1) The image 502 shown in FIG. 1 is divided into a foreground image 502a and a background image 502b to obtain Figure 7a (5) Image 506 shown.

[0360] Then, the phone Figure 7a (5) The image 506 shown is executed Figure 3b In S302 shown in FIG. 5 , a sports card 509 is instantiated on the image 506 to switch the display to Figure 7a (6) Image 506 with sports card 509 shown.

[0361] Figure 7b FIG. 4 shows a schematic diagram of the process of segmenting the image 502 by the mobile phone. Figure 7b As shown, the mobile phone can Figure 3a The S101a shown here is for Figure 7b (1) The image 502 shown is also Figure 7aThe image 502 (an example of the first image) shown in (1) is segmented to obtain Figure 7b (2) shows a foreground image 502a with a foreground element 5021 and a background image 502b with a background element 5022. In the foreground image 502a, the areas other than the foreground element 5021 (the walking person element) (e.g., the area where the background element 5022 is located) are all transparent areas. And the areas other than the background element 5022 (the window of the building where the gym is located) in the background image 502b (e.g., the area where the foreground element 5021 is located) are all transparent areas. For the convenience of the reader, Figure 7b The transparent area in the background image 502b shown in (2) is shown by a dotted line.

[0362] Figure 7c It shows that the mobile phone uses S101b and S101c to Figure 7a (4) The sports card 503 shown is processed to obtain Figure 7c The process of image 5051 (an example of the third image) is shown.

[0363] First, if Figure 7c As shown in (1), the mobile phone can Figure 7a The sports card 503 shown in (4) is converted into a picture format through S101b to obtain an image 505 (an example of a second image).

[0364] The phone can then determine Figure 7c (2) The first area of ​​the image 502 (an example of the first image) in which the service card is to be displayed. The image 502 is also Figure 7a (1) and Figure 7a (2) The image 502 shown, Figure 7c The first area 5023 is shown in a dotted box in (2).

[0365] like Figure 7c As shown in (2), the first area 5023 displays an image of a curved window glass. Therefore, the background element corresponding to the first area (here, the window glass element) has a perspective parameter, which may include a perspective direction and a perspective angle.

[0366] The perspective direction is the normal vector perpendicular to the plane, and the perspective angle is the wide angle of the photo or the focal length of the photo.

[0367] Next, please refer to Figure 7c(1), the mobile phone can adjust the shape, size, and perspective parameters of the image 505 based on the perspective parameters of the background elements of the first area 5023 and the shape and size of the first area, so that the perspective parameters of the adjusted image 5051 (an example of the third image) are consistent with Figure 7c (1) and Figure 7c (2) The perspective parameters of the background elements displayed on the first area 5023 are the same, and the shape and size of the image 5051 are also the same as the shape and size of the first area 5023.

[0368] Finally, the mobile phone can refer to the application scenario 1 Figure 6c The principle of the synthesis process shown in Figure 7c The image 5051 shown in (1) is synthesized in Figure 7b (2) The first area between the foreground image 502a and the background image 502b shown in FIG. Figure 7c (2) shown in the first region) to obtain Figure 7a (5) An image 506 (an example of a target image) is shown.

[0369] Furthermore, the mobile phone may write the perspective parameters, shape, size and related card type (here, sports card) of the adjusted image 5051 as card metadata into Figure 7a (5) In the metadata of the image 506 shown.

[0370] In this way, when a service card is added to an image, when the image elements (not limited to background elements, but also including foreground elements, for example, when there are multiple foreground images) displayed at the position where the service card is to be added to the image have a certain tilt angle, such as curved glass, the mobile phone can refer to the tilt angle (specifically the perspective direction and perspective angle) to adjust the tilt angle of the service card in picture format to be added to the image, so that the service card in picture format that is attached to the position in the image is more closely aligned with the plane of the background elements at the attachment position in the image, so that the service card in picture format added to the target image can be more integrated with the relevant foreground elements and / or background elements in the target image.

[0371] Figure 7d An example of a mobile phone pair is shown. Figure 7a (5) The first area in the image 506 shown instantiates a process of a sports card 509.

[0372] First, if Figure 7d As shown, the mobile phone can be based on Figure 7a (5) The card metadata about the image 5051 related to the image 506 shown in FIG. Figure 3bS302 shown in FIG. 1 is used to call the card creation interface of the sports health application to create a sports card 503. The sports card 503 can be used with Figure 7a (4) is the same as the sports card 503 shown in FIG. 5. The mobile phone can pass the card type and card information in the card metadata to the above interface to create the sports card 503.

[0373] Then, the mobile phone can use the card size, card shape, and card perspective parameters in the above card metadata to Figure 3b As shown in S302 , the size, shape, and perspective parameters of the sports card 503 are adjusted to obtain a second service card, which is the sports card 509 in this case.

[0374] Finally, please refer to Figure 7a (5), the mobile phone may insert the sports card 509 into the first area between the foreground image 502a and the image 5051 of the image 506 (see Figure 7c (2) shown in the first area 5023), to obtain Figure 7a (6) shows an image 506 in which a sports card 509 is instantiated in the first area.

[0375] In some embodiments, the phone can also respond to the phone clicking Figure 7a (4) The operation of the control 406 shown in FIG. 4 directly switches the display interface to Figure 7a (6) The display interface 201 shown in FIG. Figure 7a (5) shown in the image 506, however, the target image stored in the mobile phone is still Figure 7a (5) As shown in the image 506, the implementation principle of this process is similar and will not be repeated here.

[0376] In an embodiment of the present application, the mobile phone can recommend a service card that matches the content of the image according to the image of the service card to be added, so that the type of service card added to the image is more matched with the content and scene of the image. Then, when adding a service card in picture format to the image, the mobile phone can refer to the perspective parameters of the image element displayed in the first area of ​​the service card to be added, and optionally, it can also include shape, size, transparency, lighting characteristics, hue, transparency, etc., to further adjust the service card in picture format, so that the picture of the service card in picture format (i.e., the third image) after adjustment can be more closely matched with the image element related to the first area in the first image, and the sense of integration is stronger. Similarly, when instantiating a service card for a target image with a service card in picture format, the above-mentioned perspective parameters, shape, size, transparency, lighting characteristics, hue, etc. of the third image can be referred to to adjust the parameters of the corresponding type of service card created, and then the adjusted service card is attached to the first area of ​​the target image, so that the service card instantiated on the target image can be more closely matched with the image element related to the first area in the target image, and the sense of integration is stronger.

[0377] Application Scenario 3

[0378] In the above-mentioned application scenarios 1 and 2, the implementation process of the method of the present application is mainly explained by taking the example that a mobile phone can add service cards to a first image stored locally, and the added service cards are service cards selected by the user and / or recommended by the system based on the first image.

[0379] In this application scenario 3, Figure 8a , Figure 8b Take the example to describe the above Figure 3a to Figure 3b implementation process.

[0380] Most of the implementation processes of this application scenario 3 are the same as those of the above application scenarios 1 and 2, with the following three differences:

[0381] Difference 1: The mobile phone is passing Figure 3a When determining the first image of the service card to be added in S100a, the image currently captured by the mobile phone can be used as the first image.

[0382] Difference 2: The mobile phone is passing Figure 3a When determining the first service card in S100b, the mobile phone can determine the first service card according to the real-time scenario. For example, when the mobile phone captures the first image or selects the first image to add a service card, the real-time scenario of the mobile phone is that the mobile phone is playing music, and the mobile phone can recommend a music card according to the real-time scenario (playing music).

[0383] Difference 3, mobile phone through Figure 3a and Figure 3b After obtaining the target image with the service card instantiated by the process, the mobile phone can respond to the user operation on the service card displayed on the target image by refreshing the card information of the service card and / or calling the music application to play music. Of course, the instantiated service cards displayed on the target image in the above application scenarios 1 and 2 are also interactive, and the principle is the same.

[0384] Combine the following Figure 3a to Figure 3b , come on Figure 8a and Figure 8b The application scenario 3 shown is described, and the differences between the application scenario 1 and the application scenario 2 are mainly introduced, and the similarities are not repeated.

[0385] Please refer to Figure 8a ,like Figure 8a As shown in (1), the display interface 600 of the mobile phone is the application interface of the camera application, and the display interface 600 may include but is not limited to: a shooting preview interface 608, a shooting button 602, a control 603 for displaying the currently shot image, and a control 604 for switching the shooting camera (for example, switching between the rear camera and the front camera), etc. Among them, the current shooting preview interface 608 displays an image screen 601 captured by the rear camera. In addition, the applications running in the background of the mobile phone include a music application, and the music application is playing an audio named "Song 1".

[0386] like Figure 8a As shown in (1), the user clicks the shooting button 602, and the mobile phone can shoot the image screen 601 displayed on the shooting preview interface 601 in response to the user operation to obtain a first image that is the same as the image screen 601. Since the first image is the same as the image screen 601, for the sake of convenience of explanation, this embodiment uses image 601 to represent the first image.

[0387] And the mobile phone can respond to the user clicking the shooting button 602, determine that the service card to be added to the image 601 is a music card, and execute Figure 3b The above S101a, S101b, S101c, S102, S103, S104 shown in the figure create the music card and convert the music card into Figure 8a (3) and insert the image 607 into the image 601 to obtain the image Figure 8a (3) The image 605 shown in FIG. 6 is the original image of the music card in the picture format that was taken this time. The mobile phone can store the image 605 in the gallery application. The mobile phone can also display the interface from Figure 8a The display interface 600 shown in (1) switches to Figure 8a(2) The display interface 600 shown in FIG. 6 shows a shooting preview interface 608 in the display interface 600, which displays an image 601' currently captured by the camera. In addition, Figure 8a (2) The control 603 shown can display Figure 8a (3) shows a thumbnail 6031 of the image 605 obtained by the current shooting.

[0388] In order to facilitate readers to clearly see the thumbnail 6031, Figure 8a (4) shows the enlarged effect of the thumbnail 6031. Figure 8a The original image corresponding to the thumbnail 6031 shown in (3) is image 605. The thumbnail 6031 is an image obtained by cropping and reducing the image 605 according to the circular view.

[0389] The following describes the mobile phone generation Figure 8a (3) In the process shown in image 605, the mobile phone can respond to the user clicking Figure 8a (1) The user operates the capture button 602 shown in FIG. 6 to determine that the mobile phone is running a music application to play an audio file named "Song 1" (a real-time scenario); then, the mobile phone can determine that the first service card to be added to the captured image 601 is a music card, and obtain the name of the audio file being played, here "Song 1", and the playback progress of the audio file, etc., through the music application. Then, the mobile phone can call an interface for creating a music card, and pass the audio file name, audio playback progress, etc., as parameters to the interface to create a music card. The card information displayed on the music card is the same as that on the music card. Figure 8a (3) is the same as the information shown in the image 607 shown in FIG. Then, the mobile phone can convert the music card into a picture format to obtain Figure 8a (3) is a music card in the picture format shown, namely, image 607. Then, the mobile phone inserts the image 607 between the foreground image and the background image of the above image 601 to obtain Figure 8a (3) Image 605 shown.

[0390] The foreground image is generated by the mobile phone after performing a segmentation operation on the image 601 in S101a. Figure 8a The foreground element 6051 (here, the sky element) shown in the figure, and the area other than the foreground element 6051 is a transparent area; the background image is generated by the mobile phone after the image 601 is segmented by S101a. Figure 8a The image shown is a background element 6052 (here, an image element composed of multiple trees), and the area other than the background element 6052 is a transparent area.

[0391] In this way, Figure 8aAs shown in (3), the image 605 (an example of the target image) captured and processed by the mobile phone displays an image 607 (a music card in picture format), and the image 607 displays the name of the audio, here "Song 1", the poster image of the audio, and buttons for playing and controlling the audio. However, Figure 8a (3) and the image 607 inserted into the image 605 shown in FIG8 (4) cannot perform card interaction, such as audio playback control and other operations. And the mobile phone can also store the card metadata related to the image 607 in the metadata of the image 605. The card metadata may include: the card type is a music card, the card information includes the audio name "song 1", the song progress is the above-mentioned "playing progress", the poster picture and other information.

[0392] Then, if Figure 8a As shown in (2), the user clicks on the control 603, and the mobile phone can respond to the user operation to switch to Figure 8a (5) The display interface 201 shown in FIG.

[0393] Specifically, in response to the user clicking the control 603, the mobile phone can refer to the card metadata related to the image 605 to store the image in the gallery application. Figure 8a (3) The image 605 shown in FIG. 6 is used to instantiate a music card 606 to insert the music card 606 into the Figure 8a (3) is inserted between the foreground image of the image 605 (the image including the foreground element 6051 described above) and the image 607, and the insertion position coincides with the image 607 to cover the image 607, thereby obtaining the following: Figure 8a (5) Image 605 shown.

[0394] like Figure 8a As shown in (5), the instantiated music card 606 displayed on the image 605 may include but is not limited to: an audio name 6062 (here, "Song 1"), a poster image 6061 of "Song 1", a button 6063 for switching to the previous song, a button 6064 for switching to the next song, and a button 6065 for controlling the playback and pause of the audio. Figure 8a (3) The various playback control buttons shown in the image 607 are not interactive. Figure 8a The play control buttons in the music card 606 shown in (5), such as button 6063, button 6064, and button 6065, are all interactive.

[0395] In this way, the embodiment of the present application can determine the service card to be added to the first image based on the real-time scenario, so that the type of service card added to the image and the card information can match the real-time scenario.

[0396] In addition, Figure 8a In the embodiment, the mobile phone can automatically add a corresponding service card to the first image according to the real-time scene, then the gallery application or the camera application can provide a switch for whether to automatically add a service card to the image. When the switch is turned on, the mobile phone can automatically add a service card to the first image currently shot or selected by the user according to the real-time scene. When the switch is turned off, the mobile phone can add a service card to the first image according to the user operation.

[0397] It should be understood that the real-time scenario described in the present application is not limited to the real-time scenario when the image is captured, but may also be other real-time scenarios related to adding a service card to the first image.

[0398] For example, a mobile phone displays multiple windows, including window 1 for displaying a video playback interface and window 2 for adding a display interface for a service card to a first image. The real-time scenario may be information about a video being played in the video playback window in window 1. Based on the real-time scenario, the mobile phone may add a video card related to the video being played to the first image to generate and save a corresponding target image. When displaying the target image, the corresponding video card may be instantiated for the target image.

[0399] For another example, the user switches the display interface of the mobile phone from the application interface for playing videos to the application interface for adding a service card to the first image. The real-time scenario can be the information of the relevant functions (here, the video playing function) used by the application recently run by the mobile phone before adding the service card to the first image, such as the information of the recently played video or the information of the user's favorite video, etc., which is not limited to the specific information and can be different according to the application scenario. Then, based on the real-time scenario, the mobile phone can add the video card related to the user's favorite video to the first image to generate and save the corresponding target image, and then instantiate the corresponding video card for the target image when displaying the target image.

[0400] Figure 8b An example is shown Figure 8a (5) The interactive scene of the music card 606 in the image 605 shown.

[0401] like Figure 8b As shown in (1), the user clicks on the control 6065 in the music card 606 displayed on the image 605. The mobile phone can respond to the user operation and call the music application through the music card 606 to play the audio named "Song 1". For example, according to the playing progress recorded in the card metadata, the mobile phone can start playing the audio named "Song 1" from the playing progress, and as shown in FIG. Figure 8b As shown in (2), the mobile phone can change the graphic of the button 6065 in the music card 606 from Figure 8b The triangle shown in (1) is updated to Figure 8b (1) The double vertical lines shown indicate that the audio is in the playing state and can be paused.

[0402] In this way, the service card inserted in the image of the present application is an interactive service card, rather than a non-interactive picture showing card information, thereby achieving the effect of embedding the service card in the image.

[0403] Application Scenario 4

[0404] The difference between this application scenario 4 and the above-mentioned application scenarios 1 to 3 is that the added service card is a map card, and other processes can adopt the principles of the implementation process of any one or more application scenarios in the above-mentioned application scenarios 1 to 3, which will not be repeated here.

[0405] In this application scenario 4, Fig. 9 Take the example to describe the above Figure 3a to Figure 3b implementation process.

[0406] Combine the following Figure 3a to Figure 3b , come on Fig. 9 The application scenario 4 shown is described, and the differences between the application scenarios 1 to 3 described above are mainly introduced, and the similarities are not repeated.

[0407] Please refer to Fig. 9 ,like Fig. 9 As shown in (1), the display interface 201 of the mobile phone is the application interface of the gallery application. The display interface 201 displays an image 701 stored in the gallery application. The image 701 may include a foreground element 7011 (a building), a foreground element 7012 (another building) and a background element 7013 (here, environmental elements such as roads around the above two buildings).

[0408] Users can use the above Figure 5a The process shown in the figure is to select a service card to be added for the image 701 from the service cards provided by the system. Figure 7a The process shown in the figure is to select the service card to be added for the image 701 from the service cards recommended by the system that match the image scene of the image 701. The mobile phone can also Figure 8a The related embodiments describe automatically determining the service card to be added to the image 701 based on the real-time scenario.

[0409] For example, users Fig. 9(1) Perform a long press operation on the image 701 shown, which indicates adding a service card to the image 701. The mobile phone can determine the map card that matches the scene of the image 701 (here, the building scene) in response to the user operation. Then, the mobile phone can determine the actual location of the screen displayed by the image 701 by analyzing the shooting location of the image 701 or the foreground elements 7011 and 7012 in the image 701, which is the "XX Building" here. Next, the mobile phone can call the map application and pass the actual location as a parameter to the map application to create a map card related to the actual location. Then, the mobile phone can convert the map card into an image 702 in a picture format and add it to the image 701 to obtain Fig. 9 (2) The image 704 (a target image) shown is saved in the gallery application, and the card metadata associated with the image 702 is saved in the metadata of the image 704.

[0410] like Fig. 9 As shown in (2), the image 702 displays an icon 7021, and "XX Building" is displayed near the icon 7021 to indicate the actual location of the building in the image 704, and the image 702 also displays map information such as roads around the "XX Building".

[0411] And mobile phones can Fig. 9 (2) The image 704 shown in FIG. 1 is instantiated as a map card, and a map card 705 with the same card information as the image information displayed in the image 702 is added between the foreground element and the background element of the image 704, so as to Fig. 9 (2) The display interface 201 shown in FIG. Fig. 9 (3) The display interface 201 shown. Fig. 9 As shown in (3), the map card 705 is partially blocked by the foreground element 7011 and the foreground element 7012, and the map card 705 can block the background element 7013.

[0412] Next, the user clicks Fig. 9 (3) The mobile phone can respond to the user operation by calling the map application through the map card 705 to switch the display Fig. 9(4) The display interface 703 shown in the figure is a map interface of the location of the "XX Building" provided by the map application. The display interface 703 may include but is not limited to: a control 7032 for searching for a location, and a map screen 7031 of the location of the "XX Building". The map screen 7031 can perform operations such as zooming in and out of the map information and moving the map information. The present application does not limit the specific information displayed on the display interface 703, which depends on the function of the map application and is not limited here.

[0413] In this way, the mobile phone of the present application can add a map card that matches the location of the building to a picture including a building. The map card can display card information related to the location, and the map card displayed on the picture is interactive. Users can operate the map card displayed in the picture to enter the map application and perform related operations.

[0414] Example 2

[0415] In the above-mentioned example 1 and its above-mentioned application scenarios 1 to 4, the process of the method for adding a service card of the present application may include: the mobile phone first adds a service card in a picture format to a first image to obtain a target image, and when displaying the target image, instantiates a service card at the position where the service card in the picture format is displayed in the target image to obtain a target image with an interactive service card.

[0416] Different from the above-mentioned Example 1 and application scenarios 1 to 4 of Example 1, in Example 2, the electronic device can Figure 2 The application layer shown adds a target application, and the target application can execute the image processing method to directly add the instantiated service card to the selected picture, so that when the target application displays the picture, the interactive service card can be displayed in the picture.

[0417] The following describes the implementation process of the method in the above-mentioned Example 2 of the present application in combination with Application Scenario 5.

[0418] Figure 4 The schematic diagram exemplarily shows the implementation process of the method of Example 2.

[0419] Combine the following Fig.10a and Fig.10b The application scenario 5 shown in the figure is used to describe the present invention. Figure 4 The process of adding a service card is explained in the figure.

[0420] in addition, Figure 4 The process shown is the same as in Example 1 above. Figure 3aThe difference of the process shown mainly lies in that there is no need to add a service card in picture format to the first image, and the principles of other processes are similar.

[0421] like Figure 4 As shown, the process may include the following steps:

[0422] S500a: The mobile phone determines a first image.

[0423] Among them, the implementation principle of S500a is similar to Figure 3a The implementation principle of S100a shown is the same and will not be repeated here.

[0424] In application scenario 5, please refer to Fig.10a , Fig.10a The display interface 201 shown in (1) is an application interface of a gallery application, and the display interface 201 displays an image 801 saved in the gallery application. The image 801 is a picture of a user sitting on a sofa and watching a video displayed on a screen, wherein the image 801 exemplarily shows foreground elements 8011 (human elements, sofa elements), foreground elements 8012 (such as cabinet elements), and background elements 8013 (screen elements), wherein the screen elements are tilted at an angle, so that the background elements 8013 have perspective parameters.

[0425] In this application scenario 5, the user can long press Fig.10a (1) to request to add a service card to the image 801. The mobile phone can respond to the user operation by executing the following S500b, S501a, S501c, S502, S503, and S504 to instantiate a video card on the image 801 to generate a target application, and display the application icon 902 of the target application on the image 801. Fig.10a (2) On the desktop interface 901 shown.

[0426] In a specific implementation, the mobile phone can respond to the user's long press Fig.10a (1) The operation of the image 801 shown in FIG. 8 is performed by calling the compiler of the mobile phone through the gallery application to Figure 4 The code related to the process is compiled to generate a target application, which may include a target image with an interactive service card and card metadata of the service card.

[0427] In some embodiments, the mobile phone can compile the above code through multiple compilers (or one compiler) related to different operating systems to obtain multiple target applications that can run on different operating systems. The operating system may include but is not limited to Android system, IOS system, Windows system, Hongmeng system, etc.

[0428] In this way, when a user needs to share a target application with an interactive service card across operating systems, for example, if mobile phone 1 is a Hongmeng system and mobile phone 2 is an Android system, mobile phone 1 can share the target application with an interactive service card across operating systems. Figure 4 The process generates a target application 1 that can be run on the Hongmeng system, and generates a target application 2 that can be run on the Android system. Mobile phone 1 can run the target application 1 to display the target image with the interactive service card, and mobile phone 1 can share the target application 2 to mobile phone 2, so that mobile phone 2 can run the target application 2 on the Android system to display the target image with the interactive service card. In this way, the mobile phone can realize cross-platform sharing of the target application.

[0429] In some embodiments, when the computing power of mobile phone 1 is limited and it is unable to compile the target application, mobile phone 1 can interact with the cloud to enable the cloud to compile multiple target applications that can run on different operating systems, and the cloud can distribute the multiple target applications to various electronic devices. For example, the cloud can send the target application 1 to mobile phone 1 and send the target application 2 to mobile phone 2, so that mobile phone 1 can share the target application with mobile phone 2.

[0430] Since the target application 2 shared by mobile phone 1 to mobile phone 2 includes a target image with an interactive service card, mobile phone 2 can display the target image instantiated with the service card after starting the target application 2, and can interact with the service card through the target application.

[0431] Regarding the application scenarios of target application sharing, Figure 5c and Figure 5d The application scenarios of the embodiments are similar, the only difference is that the shared content is not an image but a target application, and the specific process will not be repeated here.

[0432] S500b, the mobile phone determines the first service card.

[0433] Among them, the implementation principle of S500b is the same as Figure 3a The implementation principle of S100b shown is the same and will not be repeated here.

[0434] exist Fig.10a In the application scenario 5 shown, the mobile phone can determine Fig.10a (1) The first service card that matches the image 801 shown is a video card, and the mobile phone can identify the background element 8013 in the image 801 to determine the video name, video playback progress and other information of the video watched by the person in the image 801; then, the mobile phone can call the interface of the video application and pass the above information as a parameter to the interface to create a video application. Fig.10b Video card 803 is shown.

[0435] like Fig.10b As shown, the video card 803 may include but is not limited to: video screen 8033, play pause control 8031, progress bar 8034, play progress adjustment control 8032 and other card information.

[0436] Among them, the video screen 8033 and Fig.10a The video picture shown by the background element 8013 shown in (1) corresponds to the same frame of the same video. In this way, the video card 803 created by the present application is a video card that matches the video picture displayed by the image 801 on the screen.

[0437] like Figure 4 As shown, after S500a, the method may further include S501a.

[0438] S501a, the mobile phone may divide the first image into a foreground image including foreground elements and a background image including background elements.

[0439] Among them, the implementation principle of S501a is the same as Figure 3a The implementation principle of S101a shown is the same and will not be repeated here.

[0440] exist Fig.10a In the application scenario 5 shown, the mobile phone can divide the image 801 (an example of the first image) into a foreground image including foreground elements 8011 and 8012, and other areas are transparent areas, and a background image including background elements 8013, and other areas are transparent areas.

[0441] Alternatively, if Figure 4 As shown, after S500b, the method further includes S501c.

[0442] S501c, the mobile phone can adjust the first service card to the second service card.

[0443] Among them, the implementation principle of S501c is the same as Figure 3a The implementation principle of S101c shown is the same, the only difference is that the object of adjustment is changed from an image to a service card.

[0444] In some embodiments, the mobile phone can determine a first area in the first image where a service card is inserted; and the mobile phone can refer to image parameters of the first image related to the first area to adjust corresponding image parameters of the first service card, so that after the second service card with adjusted image parameters is inserted into the first image, the second service card can be more integrated with the picture of the first image. In this way, the service card inserted into the first image can be displayed without being abrupt on the entire image.

[0445] In some embodiments, as described above, the first image of the service card to be added can be divided into a foreground image and a background image. Then, when the mobile phone determines the first area to which the service card is to be inserted, the corresponding service card can be inserted between the foreground image and the background image, and the foreground element in the foreground image can block the area ratio of the service card within a preset threshold range, for example, 3% to 20% of the area of ​​the service card is blocked by the foreground element, so as to determine the position of the first area. The preset threshold range can be flexibly changed according to actual needs and is not limited here.

[0446] In some embodiments, the mobile phone may also determine the first area based on the position or area selected by the user for the first image. For example, if the user selects a coordinate position of a service card to be added in the first image, the mobile phone may determine the first area including the coordinate position based on the coordinate position.

[0447] In some embodiments, the image parameters of the first image related to the first area may include but are not limited to: the size of the first area, the shape of the first area, the perspective parameters of the background elements located in the first area (which may include at least one of the perspective direction and the perspective angle), the color, hue, transparency, lighting characteristics and other information of the foreground elements and / or background elements related to the first area.

[0448] Among them, the solution for adjusting the first service card by the mobile phone with reference to the perspective parameters (which may include at least one of the perspective direction and the perspective angle) of the background element located in the first area will be described below. Fig.10a and Fig.10b The application scenario 5 is described in detail and will not be repeated here.

[0449] The lighting characteristics may be at least one lighting characteristic such as direct lighting, reflected lighting, shadow, etc.

[0450] For example, the background image of the first image includes a sun element, and the light emitted by the sun element shines on the human element in the foreground image, and the insertion position of the first service card in the first image (such as the first area mentioned above) is on the illumination path. Then the mobile phone can use the Ray Tracing (RT) technology to calculate the direct illumination characteristics of the area or position on the illumination path in the first service card, and render the direct illumination characteristics of the position on the illumination path in the first service card to obtain the second service card, so that after the second service card is inserted into the first area in the first image, the target image is obtained. The position on the illumination path in the second service card displayed on the target image can display a visual effect of direct illumination. Optionally, since the calculation of illumination characteristics requires a high computing power, the mobile phone can interact with the server to realize the calculation of the direct illumination characteristics of the image, so as to obtain the rendering result of the second service card, and then use the rendering result to obtain the second service card.

[0451] Of course, the way in which the mobile phone renders a visual effect with corresponding lighting characteristics on a service card is not limited to the above examples, and other technologies such as AIGC can also be used to calculate the lighting characteristics of the service card.

[0452] exist Fig.10a and Fig.10b In the application scenario 5 shown, the mobile phone can determine that the first area to be inserted into the service card is Fig.10a (1) shows the area where the background element 8013 is located. In this embodiment, the background element 8013 located in the first area is a curtain, and the curtain has a tilt angle, so that the background element 8013 has a perspective parameter.

[0453] like Fig.10b As shown, the mobile phone can Figure 4 In S501b, according to the perspective parameters of the background element 8013, the perspective parameters of the video card 803 (an example of the first service card) are adjusted, specifically including the adjustment of the perspective direction and the perspective angle, to obtain the video card 802. The video card 802 may include but is not limited to: video screen 8023, play pause control 8021, progress bar 8024, play progress adjustment control 8022 and other card information.

[0454] Since the entire video card 803 is changed into the video card 802 through the adjustment of the perspective direction and perspective angle, the perspective parameters of the video screen 8033 and the video screen 8023 are changed, the perspective parameters of the play pause control 8031 ​​and the play pause control 8021 are changed, the perspective parameters of the progress bar 8034 and the progress bar 8024 are changed, and the perspective parameters of the play progress adjustment control 8032 and the play progress adjustment control 8022 are changed. In addition, the card shapes of the video card 803 and the video card 802 are changed due to the change of the perspective parameters.

[0455] S502: The mobile phone combines a foreground image including foreground elements, a background image including background elements, and a second service card into a target image.

[0456] Among them, the implementation principle of S502 is the same as Figure 3a The implementation principle of S102 shown is the same and will not be repeated here.

[0457] like Figure 4 As shown, when the mobile phone is synthesizing, the foreground image including the foreground elements is used as the upper layer, the second service card is used as the middle layer, and the background image including the background elements is used as the lower layer to synthesize the target image.

[0458] exist Fig.10a and Fig.10b In the application scenario 5 shown, the mobile phone can Fig.10b The video card 802 shown (an example of a second service card) is inserted into Fig.10a (1) is a first area between the foreground image and the background image of the image 801, and the first area is a position corresponding to the background element 8013, so that the video card 802 can cover the background element 8013 to obtain the following Fig.10a (3) shows a target image 801 with an interactive video card 802.

[0459] Optionally, in S503, the mobile phone may determine card metadata related to the target image based on the target image, the first service card, and the second service card.

[0460] Among them, the implementation principle of S502 is the same as Figure 3a The implementation principle of S102 shown is the same and will not be repeated here.

[0461] The card metadata may include but is not limited to at least one of the following: card type, card information, card size (such as the size of the first area mentioned above), card position (such as the position of the first area in the first image), card shape (such as the shape of the first area mentioned above), and card image parameters.

[0462] The image parameters of the card are the image parameters of the second service card after the first service card is adjusted to the second service card through the above S501c, or the image parameters of the card may also be the image adjustment parameters (such as size adjustment parameters, transparency adjustment parameters, etc.) used when the first service card is adjusted to the second service card through the above S501c, wherein the image adjustment parameters may indicate information such as the adjustment direction and adjustment amount of the image parameters, and the image adjustment parameters are relative to the above image parameters, and the image adjustment parameters are relative image parameters, and the image parameters are absolute image parameters.

[0463] In some embodiments, the image parameters may include, but are not limited to, information such as color, hue, transparency, lighting characteristics, and perspective parameters.

[0464] For example, the mobile phone can determine the card type and card information of the first service card based on the first service card determined in S500b, and the mobile phone can determine the card perspective parameters, card size, card position, card color, card tone, card lighting characteristics, card transparency, etc. in the card metadata based on the perspective parameters, size and color, hue, lighting characteristics, transparency, etc. of the synthesis position (the first area mentioned above) in the first image of the second service card used when performing the synthesis operation in S502.

[0465] After S502 and S503, if Figure 4 As shown, the mobile phone may execute S504.

[0466] S504, the mobile phone may generate a target application, the target application including a target image with an interactive service card, and optionally including card metadata of the service card.

[0467] like Fig.10a As shown, the mobile phone can respond to the user's long press Fig.10a (1) The operation of the image 801 shown in FIG. 8 is performed by compiling the above-mentioned processes S500b, S501a, S501c, S502, S503, and S504 to generate a target application, and the application icon 902 of the target application is displayed on the screen. Fig.10a (2) On the desktop interface 901 shown.

[0468] When the user clicks Fig.10a (2) When the application icon 902 shown in FIG. 1 is displayed, the mobile phone can respond to the user operation and start the target application to switch the display to the following example: Fig.10a (3) The display interface 903 shown. The display interface 903 is the application interface of the target application, and the display interface 903 may include the image 801 with the interactive video card 802 stored in the target application obtained in S502, as well as the control 207 and the like.

[0469] The control 207 here is different from the control 207 involved in the above-mentioned application scenarios 1 to 4. The difference is that in the application scenario 5, Fig.10a The control 207 shown is used to indicate the sharing target application, rather than inserting an image of a service card in a picture format. The other process principles are the same and will not be repeated here.

[0470] In this embodiment, the mobile phone can instantiate a service card on the selected image without first pasting a picture of a service card on the image and then pasting the instantiated service card, which simplifies the operation process of adding a service card to the image.

[0471] Continue to refer to Fig.10a (3) The target application generated by the mobile phone may include an image 801 with a video card 802, and the video card 802 is interactive. Fig.10a As shown in (3), the user clicks the play pause control 8021, and the mobile phone can respond to the user operation by calling the video application through the video card 802 to start playing the related video displayed by the video card 802, so as to switch the display. Fig.10a (4) Display interface 903 shown.

[0472] like Fig.10a As shown in (4), the play pause control 8021 in the video card 802 displayed on the display interface 903 is changed from Fig.10a (2) The triangle shown is switched to Fig.10a (3) and the play-pause control 8021 of the switching shape still has the adjusted perspective parameters, such as Fig.10a (3) As shown, the double vertical lines shown in the play pause control 8021 are shaped with an inclined angle, while the vertical double vertical lines are shaped with an inclined angle. Fig.10a The video card 802 shown in (3) can play the video according to the recorded video playing progress to display the video playing screen.

[0473] Continue to refer to Fig.10a (4) The user slides the playback progress adjustment control 8022 in the video card 802 from the position A indicated by the dotted arrow on the progress bar 8024 to the position B indicated by the dotted arrow, so as to change the video playback progress of the video played by the video card 802. In response to the user operation, the mobile phone can call the video application through the video card 802 to adjust the playback progress of the video card 802 and switch the video screen displayed by the video card 802 to switch to the video card 802. Fig.10a (5) Video card 802 shown.

[0474] like Figure 8a (5) shows that the play progress adjustment control 8022 is located at the position B indicated by the dotted arrow, indicating that the play progress of the video played by the video card 802 has been adjusted to the play progress indicated by the position B, and the video screen 8023 displayed by the video card 802 has also been adjusted from Figure 8a The video screen 8023 shown in (4) is switched to a video screen related to the adjusted playback progress.

[0475] In this embodiment, a target image with service cards can be obtained by generating a target application, and the service cards in the target image are interactive.

[0476] It should be understood that in the above Examples 1 and 2, the technical solution of the method of the present application is illustrated by instantiating a service card in an image. When multiple service cards are instantiated on an image, the implementation principle is the same and will not be repeated here.

[0477] In addition, in the above example 2, the mobile phone can instantiate at least one service card for a single first image to generate a target application, and the target application can include a target image with the at least one service card. When the mobile phone needs to use the method of example 2 to obtain multiple target images with instantiated service cards, the mobile phone can execute the method of example 2 for each first image. Figure 4 The process shown is used to obtain multiple target applications, each of which may include a target image with an instantiated service card, wherein each target image corresponds to each first image.

[0478] This Example 2 mainly introduces the differences between Example 1 and the above Example 1. The similarities between the two examples are not introduced in Example 2. For details, please refer to the introduction of the relevant embodiments of Example 1, which will not be described here one by one.

[0479] It should be understood that the method of the present application can insert interactive service cards into the image, but the type of the service card is not limited to the weather cards, music cards, video cards, map cards, and sports cards listed in the above application scenarios. It can also be any type of service card such as time cards, picture cards, calendar cards, schedule cards, to-do cards, payment cards, search cards, etc. The implementation principles are the same and will not be elaborated here.

[0480] In the above example 1 and its application scenarios 1 to 4, and the above example 2 and its application scenario 5, the UI control added to the picture as a service card is taken as an example to illustrate the implementation process of the method of the present application.

[0481] In some embodiments, the UI control may also be a notification message in a notification bar interface or a live window in a notification bar interface. The mobile phone may receive a user operation on the notification message or live window, and the operation is used to indicate that the operated notification message or live window is added to the picture. The mobile phone may respond to the user operation to instantiate the notification message or live window in the corresponding first area of ​​the first image through the relevant method of Example 1 or Example 2 to obtain a target image, so that an interactive notification message or live window can be displayed on the target image.

[0482] For example, the interactive scenario of the notification message can be: the user can click on the notification message displayed on the target image to display the notification bar interface on the mobile phone, or the mobile phone displays the application interface of the application that pushes the notification message, and the application interface includes application information related to the notification message.

[0483] For example, the interactive scenario of the live window may be: the user clicks on the live window displayed on the target image, and the mobile phone may open a corresponding application interface related to the live window in response to the user operation.

[0484] It should be understood that only two interaction methods are listed here for example, and the present application does not limit the interaction method and its scenario. The specific interaction method depends on the interactive function of the notification message or the live window itself.

[0485] For example, the interactive scenario of a notification message can also be: in response to the user's long press operation on the notification message displayed on the target image (the operation indicates that the notification message is converted into a service card), the mobile phone converts the notification message displayed on the target image into an instantiated service card, and the card information of the service card is related to the message content of the notification message, for example, it can be some structured notification information. For example, if the notification message is a pickup notification, the card information may include information such as the pickup code, pickup address, and product type. In this way, the interactive control displayed on the target image can be refreshed from a notification message to a service card, and the user can continue to interact with the service card on the target image.

[0486] In some embodiments, the UI control may also be an interactive text box (such as a search box, etc.), an interactive button, etc.

[0487] For example, the UI control is a search box of a map application. Taking Example 1 as an example, the mobile phone can convert the search box into a picture and add it to the first image to obtain the target image; then, each time the target image is displayed, the mobile phone can display the first area of ​​the search box in the target image in picture format, call the map application to create a search box control, and instantiate the search box of the map application on the target image. In this way, the user can enter information such as the destination in the instantiated search box displayed on the target image and trigger the search operation, and the mobile phone can call the map application to search for the destination to display the map application interface of the search results of the map application about the destination.

[0488] For example, the UI control includes the text of a phone number and a dial button for initiating a voice call. Taking the solution of Example 1 as an example, the user operates the phone number of a contact in the address book, and the operation indicates that the text of the phone number of the contact and the dial button for initiating a voice call are added to the first image. In response to the user operation, the mobile phone can convert the text of the phone number and the dial button into pictures respectively, and synthesize them into one picture. The synthesized picture includes the information of the above phone number and the icon of the dial button; then, the mobile phone can add the synthesized picture to the first image to obtain the target image. When the mobile phone displays the target image, the mobile phone can instantiate a dial button of a phone application at the position where the icon of the dial button is displayed on the target image; then, the user clicks the instantiated dial button displayed on the target image, and the mobile phone can respond to the user operation, identify the information of the above phone number displayed on the target image, call the dial interface of the phone application, and pass the information of the phone number as a parameter to the interface to realize the interactive function of calling a specific contact corresponding to the specific phone number displayed on the photo. Of course, the text of the phone number in the above UI control can also be a picture of the avatar of the corresponding contact. The implementation principle is similar and will not be repeated here.

[0489] In some embodiments, the UI control may also be a floating control (such as a floating window, a floating ball, etc.) or an application interface (such as a chat interface, etc., without limitation).

[0490] Taking the solution of Example 2 as an example, the user can operate the floating control or application interface (such as a chat interface) of interest, and the operation indicates that the corresponding floating control or chat interface is added to the first image. In response to the user operation, the mobile phone can instantiate the floating control or the corresponding chat interface at the corresponding position of the first image to obtain the target application, and the target application may include a target image displaying the interactive floating control or chat interface. When the user opens the target application, the mobile phone can display the application interface of the target application, and the application interface displays the target image with the interactive floating control or the interactive chat interface.

[0491] For example, by clicking a floating control (such as a floating ball) displayed on the target image, the mobile phone can display the application interface (such as a video playback interface) corresponding to the floating ball. The video playback interface can be displayed in the application interface of the target application, or in the video application to which the video playback interface belongs (that is, the mobile phone can display the video playback interface of the video application). There is no limitation here.

[0492] For another example, when a user inputs into a chat interface displayed on the target image, the mobile phone can respond to the user operation by calling a corresponding chat application to send the input chat content.

[0493] It should be understood that the above only partially illustrates the process of adding UI controls other than service cards to pictures by the method of the present application. The principles and methods of the process of adding UI controls other than service cards to pictures by the present application can refer to the relevant descriptions of the embodiments of Examples 1 and 2 above, and will not be repeated here one by one.

[0494] It should be noted that the above Figures 3a to 10b The same reference numerals in the figures represent the same objects. Therefore, the reference numerals in the figures are not explained one by one. The reference numerals not mentioned in the figures can be referred to in Figures 3a to 10b The explanations of the same figure marks mentioned in the figure will not be repeated here.

[0495] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0496] In a possible implementation, the present application further provides an image processing device. The image processing device includes: a first display module, configured to display a target image in response to a first user operation received on a first image, wherein the target image is a first image displaying a target user interface UI control; and an output module, configured to output an interaction result related to the target UI control in response to a second user operation received on the target UI control.

[0497] In a possible implementation, the target UI control is a music control, and the music control is related to a music application running in the background when the first user operation is received.

[0498] In a possible implementation, the target UI control is related to the first image.

[0499] In a possible implementation manner, the control type of the target UI control is related to the image content of the first image.

[0500] In a possible implementation, the control information displayed by the target UI control is related to the image content of the first image.

[0501] In a possible implementation, the perspective parameter of the target UI control is the same as the perspective parameter of the target image element corresponding to the display position of the target UI control in the first image.

[0502] In a possible implementation, the first display module is specifically used to: display a second image, wherein the second image is the first image including a first UI element, wherein the first UI element is a first UI control in a picture format; display a target UI control related to the first UI control on the second image to display a target image, wherein the first UI element is covered by the target UI control.

[0503] In a possible implementation, the first UI control and the target UI control respectively display the same control information.

[0504] In a possible implementation, the image processing device further includes: a storage module, used to store the first control metadata of the first UI control in association with the second image; and a generation module, used to generate the target UI control based on the first control metadata.

[0505] In a possible implementation, the image processing apparatus further includes: a refresh module, configured to refresh the first control metadata to the target control metadata of the target UI control.

[0506] In a possible implementation, the image processing device further includes: a second display module, configured to display a target UI control related to the first UI control on the second image in response to a received display operation on the second image, so as to display the target image.

[0507] In a possible implementation, the image processing apparatus further includes: a sending module, configured to send the second image and the first control metadata associated with the second image in response to a received sharing operation on the second image.

[0508] The first display module is specifically used to: generate a target application; and display a target image in response to a received start-up operation on the target application.

[0509] The above-mentioned image processing device can be an electronic device, a chip, etc., which is not limited here.

[0510] The effects of the above-mentioned image processing device are the same as the relevant effects of the above-mentioned method embodiment, and will not be repeated here.

[0511] In one example, Fig.11 A schematic block diagram of a device 300 according to an embodiment of the present application is shown. The device 300 may include: a processor 301 and a transceiver / transceiver pin 302 , and optionally, a memory 303 .

[0512] The components of the device 300 are coupled together via a bus 304, wherein the bus 304 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, all buses are referred to as bus 304 in the figure.

[0513] Optionally, the memory 303 may be used for instructions in the aforementioned method embodiment. The processor 301 may be used to execute instructions in the memory 303, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0514] The apparatus 300 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0515] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0516] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

[0517] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0518] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the image processing method in the above-mentioned method embodiments.

[0519] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0520] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0521] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0522] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0523] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0524] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0525] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0526] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0527] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device. Of course, the processor and the storage medium can also be present in a network device as discrete components.

[0528] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0529] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: In response to receiving a first user operation on a first image, displaying a target image, wherein the target image is a first image that displays a target user interface UI control; In response to receiving a second user operation on the target UI control, an interaction result related to the target UI control is output.

2. The method according to claim 1, characterized in that The target UI control is a music control, and the music control is related to a music application running in the background when the first user operation is received.

3. The method according to claim 1, characterized in that: The control type of the target UI control is related to the image content of the first image.

4. The method according to any one of claims 1 to 3, characterized in that The control information displayed by the target UI control is related to the image content of the first image.

5. The method according to any one of claims 1 to 4, characterized in that: The perspective parameter of the target UI control is the same as the perspective parameter of the target image element in the first image corresponding to the display position of the target UI control.

6. The method according to any one of claims 1 to 5, characterized in that The display target image comprises: Display a second image, wherein the second image is the first image including a first UI element, wherein the first UI element is a first UI control in a picture format; A target UI control related to the first UI control is displayed on the second image to display a target image, wherein the first UI element is covered by the target UI control.

7. The method according to claim 6, characterized in that The first UI control and the target UI control each display the same control information.

8. The method according to claim 6 or 7, characterized in that: The method further comprises: storing the first control metadata of the first UI control in association with the second image; The target UI control is generated based on the first control metadata.

9. The method according to claim 8, characterized in that The method further comprises: The first control metadata is refreshed as the target control metadata of the target UI control.

10. The method according to any one of claims 6 to 9, characterized in that The method further comprises: In response to the received display operation on the second image, a target UI control related to the first UI control is displayed on the second image to display the target image.

11. The method according to any one of claims 8 to 10, characterized in that The method further comprises: In response to the received sharing operation on the second image, the second image and the first control metadata associated with the second image are sent.

12. The method according to any one of claims 1 to 5, characterized in that The display target image comprises: Generate target application; In response to the received launch operation on the target application, a target image is displayed.

13. An electronic device, characterized in that: The electronic device comprises: A first display module, configured to display a target image in response to a received first user operation on a first image, wherein the target image is a first image that displays a target user interface UI control; An output module is used to output an interaction result related to the target UI control in response to a received second user operation on the target UI control.

14. An electronic device, characterized in that: include: a memory and a processor, the memory and the processor being coupled; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the image processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The invention comprises a computer program, which, when being executed on an electronic device, enables the electronic device to execute the image processing method according to any one of claims 1 to 12.

16. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the image processing method according to any one of claims 1 to 12.

17. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the image processing method described in any one of claims 1 to 12.

Images