Music playing interface display method and system and storage medium

Abstract

The invention provides a music playing interface display method and system and a storage medium, and belongs to the technical field of audio and video. A music playing interface comprises a foreground image layer, a background image layer and a song related image layer, and the method comprises the steps that a music client provides a player style setting interface, a target image uploaded by a user is sent to a server side through the setting interface, the server side carries out separation processing on the background and the foreground of the target image, and the background and the foreground of the target image are obtained; the method comprises the following steps: acquiring a target image in a background image layer to obtain first foreground image layer information of the target image, sending the first foreground image layer information to a music client, determining a first foreground image by the music client based on the first foreground image layer information and the target image, displaying the target image on the background image layer by the music client, and displaying the first foreground image on the foreground image layer. And displaying the song-related layer between the background layer and the foreground layer. According to the method, the personalized self-definition capability is provided for the user, so that the music playing interface is personalized, and the richness of the music playing interface is improved.

Description

Technical Field

[0001] The present application relates to the field of audio and video technology, and in particular to a display method, system and storage medium for a music playback interface. Background Art

[0002] With the development of computer technology and network technology, music clients are becoming more and more popular, and users can play music anytime and anywhere through music clients. During the music playing process, the music playing interface of the music client includes lyrics, singers and other contents, and the contents are relatively simple. Summary of the invention

[0003] The embodiment of the present application provides a display method, system and storage medium for a music playback interface, which can solve the problem of single display content. The technical solution is as follows:

[0004] On the one hand, a method for displaying a music playing interface is provided, the method being applied to a music client, the music playing interface comprising a foreground layer, a background layer and a song related layer, the method comprising:

[0005] The music client provides a player-style setting interface for sending the target image uploaded by the user to the server;

[0006] The server performs separation processing on the background and foreground of the target image to obtain first foreground layer information of the target image, and sends the first foreground layer information to the music client;

[0007] The music client determines a first foreground image based on the first foreground layer information and the target image;

[0008] The music client displays the target image on the background layer, displays the first foreground image on the foreground layer, and displays the song-related layer between the background layer and the foreground layer.

[0009] In an optional manner, displaying the first foreground image on the foreground layer includes:

[0010] The music client displays the first foreground image in a target area in the foreground layer, or displays a non-completely transparent portion of the first foreground image in the target area, wherein the target area covers a foreground area corresponding to the target image in the background layer.

[0011] In this way, the foreground image can completely cover the corresponding foreground area in the original image, so that two foreground images will not be displayed.

[0012] In an optional manner, the first foreground layer information includes mask information;

[0013] The music client determines the first foreground image based on the first foreground layer information and the target image, including:

[0014] The music client performs mixing processing on the target image using the mask information to obtain the first foreground image.

[0015] In this way, by using the mask information, not only the amount of data transmission between the server and the music client is small, but also the calculation method using the mask information is simple, so that the music client can quickly obtain the foreground image.

[0016] In an optional manner, the music client performs a mixing process on the target image using the mask information to obtain the first foreground image, including:

[0017] The music client determines the alpha value of the pixel at the corresponding position in the target image using the value of each position in the mask information, and combines the RGB (red green blue) value of each pixel in the target image with the alpha value to obtain the pixel value of each pixel in the first foreground image; or

[0018] The music client uses the numerical value of each position in the mask information to determine the alpha value of the corresponding pixel in the target image, combines the RGB value of each pixel in the target image with the alpha value to obtain the pixel value of each pixel in the second foreground image, and uses the pixel value of each pixel in the second foreground image to amplify the second foreground image to obtain the first foreground image.

[0019] In this way, the first foreground image is represented by RGB values ​​and alpha values, indicating that the pixel sizes of the first foreground image and the target image are the same, making it easier for the music client to correspond the target image and the first foreground image when displaying them.

[0020] In an optional manner, the server performs separation processing on the background and foreground of the target image to obtain first foreground layer information of the target image, including:

[0021] The server performs target type recognition on the target image to obtain a type recognition result; the server performs separation processing on the target image based on the type recognition result to obtain first foreground layer information containing content of the target type; or

[0022] The server inputs the target image into a U-shaped network (U-Net) to obtain first foreground layer information containing the target type of content, wherein the U-Net is used to identify the target type of content;

[0023] The target type includes a person type and / or an animal type.

[0024] In this way, multiple ways are provided to obtain foreground layer information.

[0025] In an optional manner, the method further includes:

[0026] When the proportion of the occluded area to the area of ​​the song-related layer exceeds the target threshold, the music client displays the foreground layer between the song-related layer and the background layer, wherein the occluded area is the area of ​​the song-related layer obscured by the foreground layer.

[0027] In this way, when the foreground image blocks the song-related layers more often, in order not to affect the display of the song-related layers, the song-related layers are set above the foreground layers.

[0028] In an optional manner, the method further includes:

[0029] The music client selects one of the foreground layer, the background layer and the song-related layer as a target layer in response to the layer selection instruction;

[0030] The music client adjusts the target layer in response to the instruction to adjust the target layer.

[0031] In this way, the images in the layer and the positions of the layers can also be adjusted, thereby improving the user experience.

[0032] In an optional manner, the adjustment process includes adjusting the position and size of the image within the layer or adjusting the position between layers.

[0033] In an optional manner, the method further includes:

[0034] In response to receiving the adjustment instruction of the target image, the music client performs a first operation indicated by the adjustment instruction on the target image in the background layer;

[0035] The adjustment instruction includes an instruction for adjusting size, an instruction for adjusting position or an instruction for adjusting style.

[0036] This way, you can also adjust the image in the background layer.

[0037] In an optional manner, the method further includes:

[0038] The music client determines, in response to performing the first operation on the target image, a second operation to be performed on the first foreground image, wherein the second operation corresponds to the first operation;

[0039] The second operation is performed on the first foreground image in the foreground layer.

[0040] In this way, after adjusting the image in the background layer, the foreground image is adjusted so that the image in the foreground layer does not conflict with the image in the background layer.

[0041] In an optional manner, the song-related layer includes a record player icon and / or a song icon;

[0042] The method further comprises:

[0043] The music client controls the rotation of the record player icon and / or song icon in the song-related layer.

[0044] In an optional manner, the music playing interface further includes a user interface layer;

[0045] The method further comprises:

[0046] The music client displays the user interface layer on the top layer of the music playing interface.

[0047] In this way, setting the user interface layer allows users to manipulate the music playback interface, thereby improving the user experience.

[0048] In an optional manner, the method further includes:

[0049] The music client drives the first foreground image in the foreground layer to change according to the rhythm information, wherein the rhythm information is the rhythm information of the music currently played by the music client.

[0050] In this way, the foreground image can also change according to the rhythm information, thereby improving the user experience.

[0051] In an optional manner, the rhythm information includes one or more of sound loudness, frequency information or timbre sensitivity information.

[0052] In an optional manner, the method of displaying the first foreground image behind the foreground layer includes:

[0053] The music client sends a first request to the server, wherein the first request includes the target image;

[0054] The server performs super-resolution processing on the target image to obtain a super-resolution image, and separates the background and foreground of the super-resolution image to obtain second foreground layer information of the super-resolution image;

[0055] The server sends the second foreground layer information and the super-resolved image to the music client;

[0056] The music client determines a third foreground image based on the second foreground layer information and the super-resolved image;

[0057] The music client displays the super-resolved image on the background layer, and displays the third foreground image on the foreground layer.

[0058] In this way, when the image is not clear, the clarity of the image can be adjusted so that the clarity of the displayed image is higher.

[0059] In an optional manner, the music client displays the super-resolved image on the background layer and displays the third foreground image before the foreground layer, and the method further includes:

[0060] In response to receiving the operation instruction for the target image, the music client performs the operation indicated by the operation instruction on the target image, wherein the operation instruction includes one or more of a move instruction, a crop instruction or an edit instruction;

[0061] The music client performs an operation corresponding to the operation on the super-resolved image and the third foreground image.

[0062] In this way, the image can also be adjusted during the process of adjusting the clarity of the image.

[0063] In an optional manner, the music client provides a player-style setting interface for sending the target image uploaded by the user to the server, including:

[0064] The music client provides a player-style setting interface for providing super-resolution options after receiving a target image uploaded by a user;

[0065] In response to receiving the trigger instruction of the super-resolution option, the music client sends a second request to the server, wherein the second request includes the target image;

[0066] Before the server performs separation processing on the background and foreground of the target image, the method further includes: the server performs super-resolution processing on the target image;

[0067] After the server performs separation processing on the background and foreground of the target image, the method further includes: sending the super-resolved image to the music client;

[0068] The music client uses the super-resolved image as the target image.

[0069] In this way, when the image is not clear, the clarity of the image can be adjusted so that the clarity of the displayed image is higher.

[0070] On the other hand, a display system for a music playing interface is provided, the display system comprising a terminal and a server;

[0071] The terminal is installed with the music client described in the previous aspect or any optional manner of the previous aspect;

[0072] The server is installed with the server side described in the previous aspect or any optional manner of the previous aspect.

[0073] On the other hand, the present application provides a computer-readable storage medium, which stores at least one program instruction, and the program instruction is loaded and executed by a processor to implement the operations performed by the method for displaying a music playback interface as described in the previous aspect or any optional method of the previous aspect.

[0074] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0075] The music client provides the function of users uploading images. The server separates the foreground and background of the images uploaded by the user so that the music client can obtain the foreground image. In the music playback interface, the music client displays the foreground image in the foreground layer, displays the images uploaded by the user in the background layer, and displays the song-related layers between the background layer and the foreground layer. In this way, the user is provided with personalized customization capabilities, so that the music playback interface can display the images uploaded by the user, and when displaying the image, the foreground image and the image are displayed on different layers, which increases the three-dimensional effect, thereby making the content of the music playback interface richer. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0077] Figure 1 is a flow chart of a method for displaying a music playback interface provided by an embodiment of the present application;

[0078] Figure 2 is a schematic diagram of a player style setting interface provided in an embodiment of the present application;

[0079] Figure 3 is a schematic diagram of pixel values ​​of a mask image provided in an embodiment of the present application;

[0080] Figure 4 is a schematic diagram of a mask diagram provided in an embodiment of the present application;

[0081] Figure 5 is a schematic diagram of adjusting a foreground image provided by an embodiment of the present application;

[0082] Figure 6 It is a schematic diagram of kneading and shearing marks provided in an embodiment of the present application;

[0083] Figure 7 is a schematic diagram of a display interface of a music client during an image super-resolution process provided by an embodiment of the present application;

[0084] Figure 8 Schematic diagram of the principle of image super-resolution provided in an embodiment of the present application;

[0085] Fig. 9 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0086] Fig.10 It is a schematic diagram of the structure of the server provided in the embodiment of the present application. DETAILED DESCRIPTION

[0087] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0088] With the development of computer technology and network technology, music clients are becoming more and more popular. Users can play music anytime and anywhere through music clients. During the music playing process, the music client displays a music playing interface, which is the interface displayed during the music playing process. The music playing interface includes lyrics, singers and other content, which is relatively simple and does not support user customization, and has poor flexibility.

[0089] Based on this, an embodiment of the present application provides a method for displaying a music playback interface, which is a music playback interface in a music client, and the music playback interface includes a foreground layer, a background layer, and a song-related layer, etc. In this method, the music client provides a player-style setting interface, through which the image uploaded by the user can be uploaded to the server, and the server separates the foreground and background of the image and returns the foreground layer information to the music client. The music client uses the foreground layer information and the target image to determine the foreground image, and the music client displays the target image in the background layer, the foreground image in the foreground layer, and the song-related layer between the background layer and the foreground layer. In this way, the music playback interface can not only display the image uploaded by the user, but also display the foreground and background in the image in layers, with a better three-dimensional sense, which not only provides the possibility for users to customize the content of the music playback interface, but also makes the content richer.

[0090] The executors of the method are the music client and the server. The music client is a music application installed on the user's terminal, which includes a mobile phone, tablet, laptop or desktop computer, etc. The server is a program that interacts with the music client.

[0091] The music client includes a music playing interface, which is the interface of the music client when playing music. The music playing interface includes a foreground layer, a background layer, a song-related layer and a user interface (UI) layer, wherein the background layer is placed at the bottom layer, the background layer and the song-related layer are placed in the middle layer, and the UI layer is placed at the top layer. Optionally, the song-related layer is used to display content related to the song and / or the relevant content of the record player, such as the album avatar, the singer's avatar, or the relevant pictures of the song. In one example, the song-related layer includes a record player icon and / or a song icon, and the record player icon is also called a record icon, which is an icon of a vinyl record. The song icon can be an album avatar, a singer avatar, or a relevant picture of the song, which can be an image in the audio of the song, etc. The album icon can be displayed in the center of the record icon.

[0092] The following describes the process of the method for displaying the music playback interface. Figure 1 As shown, the processing steps of the method refer to step 101 to step 104.

[0093] Step 101: The music client provides a player-style setting interface for sending the target image uploaded by the user to the server.

[0094] In this embodiment, when the music playback interface is displayed, a setting option is displayed in the music playback interface. If the user wants to set the music playback interface, the setting option can be triggered, and the music client will jump to the transfer interface, which includes player-style setting options, sharing options, timed shutdown and song information. The user can click the player-style setting option, and the music client will jump to the next setting interface. The next setting interface includes viewing options, custom options, vinyl options or classic options of the music playback interface being used. The user can click the viewing option to make the music client jump to the player-style setting interface. Alternatively, a player-style setting option is displayed on the music playback interface, and the user clicks the setting option to make the music client jump directly to the player-style setting interface.

[0095] like Figure 2 As shown, the player-style setting interface includes an image adding option. The user can click the image adding button to pull up the local photo album. The user can select an image, which is called the target image. The music client sends the target image to the server.

[0096] Optionally, after selecting the target image, the user can adjust the target image here. For example, the size of the target image can be adjusted so that the size of the target image better matches the music playback interface, or the position of the target image can be adjusted so that the foreground in the target image is displayed at a desired position (such as the position of the record player icon), or the style of the target image can be adjusted. After the user completes the adjustment, the music client sends the adjusted target image to the server.

[0097] Here, when the music playback interface is displayed, the music is in the playback state or the paused state, that is to say, we can set the music playback interface when the music is playing or not playing.

[0098] In an alternative approach, see Figure 2 , in the player style setting interface, the setting options for song-related layers are displayed.

[0099] 1. In this setting interface, there are multiple styles of turntable bottom plates (i.e. the styles of vinyl bottom plates), including but not limited to: no bottom plate style, no bottom plate with pendulum style, and with bottom plate with pendulum style. Figure 2 From left to right, the vinyl floor includes the style without baseboard, the style without baseboard with pendulum needles, and the style with baseboard with pendulum needles. Users can click to select. After selecting, the music client will display the effect in the setting interface.

[0100] Optionally, image adding options and multiple styles of phono decks can be displayed in one row.

[0101] 2. In this setting interface, options for the record player color (i.e. the vinyl color) and vividness are displayed, and users can make their choices.

[0102] 3. In the setting interface, multiple styles of record player icons (i.e., vinyl styles) are displayed, such as simple translucent and complex translucent, etc. The user can click to select a style, and the music client will display the effect in the setting interface.

[0103] In an alternative approach, see Figure 2 In the player style setting interface, multiple English font styles and multiple text formats are displayed to allow users to select the text format and English font style of the music playback interface. Users can click to select. The text format includes but is not limited to left alignment and centering.

[0104] In an alternative approach, see Figure 2 In the player style setting interface, multiple styles of button combinations are also displayed, and users can click to select. The buttons included in the button combination are play control buttons. The button combinations include but are not limited to simple, rich and complete. Simple has fewer buttons than rich, and rich has fewer buttons than complete.

[0105] In an alternative approach, see Figure 2 In the player style setting interface, multiple styles of play control buttons are also displayed, and the user can click to select. The play control buttons include but are not limited to pause, fast forward, rewind or loop buttons.

[0106] In an alternative approach, see Figure 2 In the player style setting interface, a variety of light effect styles are displayed, and users can click to select. When a solid color background is displayed, users can select a light effect style. After the user selects to upload a target image, the light effect style option may not be displayed in the setting interface.

[0107] In an alternative approach, see Figure 2 In the player-style setting interface, the adjustment method of the blur and transparency of the target image added by the user is also displayed, and the adjustment method includes but is not limited to a progress bar and the like.

[0108] In an alternative approach, see Figure 2 Users can also select only a background image without displaying it in layers, select a background color from the options under "Background", or choose to add a background image, or select a background image provided by the music client.

[0109] In an alternative approach, see Figure 2 , Figure 2The current style A is also displayed in the music player. When the user chooses to adjust the style A, the adjustment effect is displayed in the style A. After the user has finished setting the music playback interface, the user can click the Done option in the upper right corner, and the music client returns to display the music playback interface.

[0110] Figure 2 The content displayed in is only an example, and the present application embodiment is not limited thereto. For example, only Figure 2 Part of the content.

[0111] Step 102: the server separates the background and foreground of the target image to obtain the first foreground layer information of the target image, and sends the first foreground layer information to the music client.

[0112] The first foreground layer information may include mask information and foreground position information. The mask information is also called a mask image, which is usually a grayscale image. The size of the mask image is the same as the size of the target image, that is, the pixel resolution is the same. In some cases, the value of each pixel in the mask image is 255 or 0, and the mask image is a black and white image. The pixel value is shown in Figure 3 (a) in the figure, the mask image can be found in Figure 4 , the white area is the foreground, and the black area is the background. In other cases, in the mask image, in order to have a smoother transition, the values ​​of each pixel at the junction of the background and foreground of the target image are not necessarily 255 or 0. Figure 3 (b) in which the value of each pixel is 0 to 255, 0 represents black, 255 represents white, and in another solution, white can also be represented by 1.

[0113] In this embodiment, after the server receives the target image, there are multiple ways to separate the foreground and background of the target image. Two feasible ways are provided below:

[0114] Method 1: For different foreground types, the server uses different cutout models to separate the foreground and background. The cutout model can be a U-Net that can recognize some foreground types, or other neural network models. The server receives the target image, identifies the target type of the target image to obtain a type recognition result, and obtains the cutout model corresponding to the type recognition result, inputs the target image into the cutout model, and the cutout model outputs the first foreground layer information. For example, the target type includes a human type and / or an animal type. If the type recognition result is a human type, the cutout model corresponding to the human type is obtained, the target image is input into the cutout model, and the cutout model outputs the first foreground layer information. The first foreground layer information is the foreground layer information containing the human type content.

[0115] Method 2: Considering that the images uploaded by users are complex and diverse, the most common ones are images of people and animals. When the cutout model recognizes images of people and animals, the effect may be better, but it cannot correctly recognize some scenes that are not real people, and for some scenes with complex composition and unclear foreground subjects (such as images with both blurred portraits and objects), the current cutout model may not be able to correctly recognize the foreground. Therefore, in an embodiment of the present application, an end-to-end U-Net can be used to separate the foreground and background. The architecture of U-Net includes Encoder and Decoder. Among them, the encoder is responsible for gradually reducing the resolution of the input image while extracting high-level semantic features. It is usually composed of multiple convolutional layers and pooling layers, each convolutional layer uses a 3x3 convolution kernel, the activation function is usually a ReLU (rectified linear unit) function, and the pooling layer is used to reduce the size of the feature map. The decoder is symmetrical with the encoder and is responsible for gradually upsampling the feature map to restore it to the same size as the original image. After each upsampling operation in the decoder, it is concatenated with the feature map of the corresponding layer in the encoder to retain more spatial information and details. In addition, an important feature of U-Net is the Skip Connection, which directly connects the feature map of the same resolution in the encoder with the feature map in the decoder. This connection method helps to preserve low-level features, thereby improving the precision and accuracy of segmentation.

[0116] The server inputs the target image into U-Net, and the output of U-Net is a matting mask image, that is, the first foreground layer information is obtained. The foreground indicated by the first foreground layer information includes but is not limited to the information of the target type of content, such as non-human content, and some other objects. The matting mask image is a type of mask image. Compared with the mask image, the foreground and background have a smoother transition. U-Net is used here, and there is no need to distinguish between portraits and objects first, which can shorten the time for separating the foreground and background and shorten the time for processing a single image to less than 1 second.

[0117] After obtaining the first foreground layer information, the server sends the first foreground layer information to the music client.

[0118] In an optional manner, in some cases, the mask image output by U-Net may have some jagged edges at some edge positions. For example, there are some white edges at the edge of the hair. If such white edges happen to be in the area where the song-related layers are superimposed, it will affect the user experience. Therefore, before obtaining the first foreground layer information, in order to make the display effect of the music client better, the first foreground layer information is post-processed. There are many ways to post-process. Two possible ways are provided below. The two possible ways can be used at the same time, or only one of them can be used.

[0119] 1. Filter out edges with low confidence based on a certain threshold.

[0120] First, the confidence is calculated: the alpha value corresponding to the mask image output by U-Net is between [0,1]. The confidence can be the alpha value itself or an enhanced alpha value obtained in some way, such as through local contrast enhancement.

[0121] Next, edge filtering is performed: a suitable threshold is selected, which can be an empirical value, such as 0.3. For edges below the threshold, its transparency can be set to 0, that is, completely transparent, or its transparency can be reduced to a lower value, so that it will not be so obvious when synthesizing the edge. Then the updated alpha value is mapped to the mask image to obtain the updated mask image, so that the first foreground layer information includes the updated mask image.

[0122] 2. Soften the edges based on bilateral filter.

[0123] Bilateral filtering is a nonlinear filtering method that reduces noise while retaining edge information. It takes into account the spatial distance and color similarity between pixels. The bilateral filter has two main parameters, one is the spatial parameter (controls the size of the filter window) and the other is the color parameter (controls the weight of color similarity). The spatial parameter and color parameter can be set according to empirical values, such as 75 for the spatial parameter and 75 for the color parameter.

[0124] The server side applies a bilateral filter to the filtered mask image to soften the edge and obtain a filtered mask image, so that the first foreground layer information includes the updated mask image.

[0125] Here, post-processing is performed on the server side. In another implementation, the music client performs the above post-processing on the mask image.

[0126] The above is explained using the example that the first foreground layer information includes a mask image. In another implementation, the first foreground layer information includes the alpha value of each pixel of the target image. In this way, the music client does not need to determine the alpha value based on the mask image.

[0127] Step 103: The music client determines the first foreground image based on the first foreground layer information and the target image.

[0128] In this embodiment, the music client receives the first foreground layer information. When the first foreground layer information includes a mask image, the music client performs alpha blending processing on the target image using the mask image to obtain the first foreground image.

[0129] In an optional manner, each pixel of the image is represented by an RGBA (red green blue alpha) value, where the value of A is called an alpha value, and the alpha value is 0 to 1, reflecting transparency. When the value is 0, it is completely transparent, when the value is 1, it is completely opaque, and when the value is between 0 and 1, it is not completely transparent. The closer the value is to 0, the more transparent it is. The music client uses the pixel value of each pixel in the mask image to determine the alpha value of the pixel at the corresponding position in the target image, and combines the RGB value of each pixel in the target image with the alpha value to obtain the RGBA value of each pixel in the first foreground image. For example, in the mask image, when the pixel value of a certain pixel is 0, the alpha value of the pixel at the corresponding position in the target image is set to 0, when the pixel value of a certain pixel is 255, the alpha value of the pixel at the corresponding position in the target image is set to 1, and when the value of a certain pixel is a value P between 0 and 255, the alpha value of the pixel at the corresponding position in the target image is set to P / 255. Like this, the first foreground image has the same size as the target image, but the background part is fully transparent or partially transparent. In this way, the first foreground image has the same size as the target image, and the position better corresponds when displaying.

[0130] In another optional method, the music client uses the numerical value of each pixel in the mask image to determine the alpha value of the pixel at the corresponding position in the target image, combines the RGB value of each pixel in the target image with the alpha value, obtains the pixel value of each pixel in the second foreground image, and uses the pixel value of each pixel to enlarge the second foreground image to obtain the first foreground image, so that the size of the foreground area in the first foreground image is larger than the size of the foreground in the target image.

[0131] When the first foreground layer information includes an alpha value, the RGB value of each pixel in the target image is directly combined with the alpha value to obtain the pixel value of each pixel in the first foreground image. In this case, the first foreground image can also be enlarged to obtain an enlarged first foreground image.

[0132] In the case where the first foreground layer information includes the position information of the foreground, the music client uses the position information to obtain the pixel values ​​of each pixel point in the area indicated by the position information in the target image to obtain the first foreground image. In this way, the size of the first foreground image is different from that of the target image, and a position correspondence relationship between the first foreground image and the target image needs to be established so that the first foreground image covers the foreground area in the background layer.

[0133] Step 104, the music client displays the target image on the background layer, displays the first foreground image on the foreground layer, and displays the song-related layer between the background layer and the foreground layer.

[0134] In this embodiment, the music client displays the target image on the background layer according to the pixel value of each pixel point in the target image, and displays the first foreground image on the foreground layer according to the pixel value of each pixel point in the first foreground image, and displays the song-related layer between the background layer and the foreground layer, see Figure 5 In addition, Figure 2 A preview diagram is also provided at the top. After the user confirms that it can be used, he can click the use option to make the music client jump to the music playback interface.

[0135] It should be noted that the target image may be displayed directly, or the foreground area in the target image (the foreground area is the area where the foreground of the target image is located) may be cut out and then displayed.

[0136] In an optional manner, in step 104, in order to prevent the foreground of the target image in the background layer from being displayed, the first foreground image is used to block the foreground. When the first foreground image is the same size as the target image, and the first foreground image and the target image are displayed overlapping, the music client displays the non-completely transparent portion of the first foreground image in the target area in the foreground layer, and the target area covers the foreground area corresponding to the target image in the background layer. The foreground area is the area where the foreground of the target image is displayed in the background layer. For example, when the mask image is a black and white image, the foreground area is the area where the specified pixel point in the target image is displayed, and the specified pixel point is the pixel point in the target image corresponding to the mask image with a pixel value of 255.

[0137] In the case where the first foreground image only includes pixels of the foreground in the target image, the music client displays the first foreground image in the target area in the foreground layer, and the target area covers the foreground area corresponding to the target image in the background layer.

[0138] In an optional manner, in step 104, when displaying the layers, the music client may directly display the song-related layers between the foreground layers and the background layers, or the music client determines the area of ​​the song-related layers blocked by the foreground layers, which is referred to as the blocking area. When the blocking area accounts for an area of ​​the song-related layers that is less than or equal to the target threshold, the music client displays the song-related layers between the background layers and the foreground layers. When the blocking area accounts for a proportion of the area of ​​the song-related layers that exceeds the target threshold, the music client displays the foreground layers between the song-related layers and the background layers. Here, the target threshold may be set based on an empirical value, such as a target threshold of 85%, or the user may set the target threshold by themselves.

[0139] In an optional manner, in order to make it more convenient for users to manage the music playing interface, after displaying the layers in step 104, the music client also provides a method for adjusting the positional relationship between the layers or the images in the layers.

[0140] In one solution, the music client also displays a UI layer, and the UI layer is displayed on the top layer. Adjustment options corresponding to each layer are displayed on the UI layer, or an adjustment option is set, but the selection methods of different layers are different. The user can select a specified layer by triggering the adjustment option, and the music client responds to the layer selection instruction and selects one of the foreground layer, background layer and song-related layer as the target layer. After selecting the target layer, the user can operate the target layer. Specifically, the user inputs the adjustment instruction of the target layer, and the music client adjusts the target layer. Assuming that the image in the target layer is an A image, the adjustment process includes adjusting the position of the A image in the target layer, and adjusting the size of the A image, or adjusting the positional relationship between the target layer and other layers. For example, the user enlarges the A image, or the user moves the target layer to the previous layer or the next layer, etc.

[0141] In another optional method, in order to facilitate users to manage the music playback interface, after the layers are displayed in step 104, an adjustment option is displayed on the UI layer. The adjustment option is used to adjust the upper and lower position relationship between the foreground layer and the song-related layers. After the user triggers the adjustment option, if the foreground layer is displayed above the song-related layers, the foreground layer is moved below the song-related layers. If the foreground layer is below the song-related layers, the background layer is moved above the song-related layers.

[0142] In one example, if Figure 5 As shown, the adjustment option changes with the position of the foreground layer. For example, if the foreground layer is above the song-related layer, the adjustment option is displayed as a downward arrow. When the user clicks the adjustment option, the music client displays the background image below the song-related layer. If the foreground layer is below the song-related layer, the adjustment option is displayed as an upward arrow. When the user clicks the adjustment option, the music client displays the background image above the song-related layer.

[0143] In another example, the adjustment option changes with the position of the song-related layer. For example, if the song-related layer is below the foreground layer, the adjustment option is displayed as an upward arrow, and the user clicks the adjustment option, and the music client displays the song-related layer above the background image. If the song-related layer is above the foreground layer, the adjustment option is displayed as a downward arrow, and the user clicks the adjustment option, and the music client displays the song-related layer below the background image.

[0144] In an optional manner, the above text describes that the target image can be adjusted when the user uploads the image. In order to provide a more convenient adjustment function, after step 104, the music client can display various adjustment buttons for adjusting the target image. The adjustment buttons are buttons for adjusting the size, adjusting the position, or adjusting the style. If the user makes some adjustments to the target image, the adjustment button corresponding to the adjustment is clicked. After the music client receives the adjustment instruction, it performs the operation indicated by the adjustment instruction on the target image. This operation is called the first operation. The music client displays the adjusted target image in the background layer. For example, the target image is enlarged, and the music client displays the enlarged target image in the background layer.

[0145] Optionally, after adjusting the target image, in order to make the first foreground image correspond to the target object, the first foreground image is also adjusted accordingly. The process is as follows:

[0146] After the music client performs the first operation on the target image, it determines the operation to be performed on the first foreground image, which is called the second operation. The second operation corresponds to the first operation, so that after the first operation is performed on the target image, the corresponding operation can also be performed on the first foreground image. The music client performs the second operation on the first foreground image in the foreground layer. For example, the first operation is an operation of enlarging the target image. After the target image is enlarged, the foreground area is also enlarged. In order to make the first foreground image cover the foreground area corresponding to the target image in the background layer, the first foreground image is also enlarged in proportion. For another example, the first operation is an operation of changing the style of the target image. After the style of the target image is changed, the first foreground image is changed to the same style.

[0147] Here, the second operation is directly performed on the first foreground image. In another process, the first foreground image is re-determined using the first foreground layer information and the adjusted target image, and the re-determined first foreground image is displayed in the foreground layer.

[0148] Optionally, resizing may be done without buttons, but through gestures. Figure 6 As shown, a pinch-to-cut mark is displayed in the setting interface, and the user can crop the displayed image by a pinch gesture. A completion option can also be displayed. After cropping, the user clicks the completion option, so that the music client displays the cropped image.

[0149] In an optional manner, the music client controls the player icon and / or the song icon to rotate in the song-related layer. In one example, the rotation speed may be a default value, and in another example, the rotation speed may be configured by the user, for example, see Figure 2 In the player-style settings interface, there is a turntable speed adjustment option, where the user can select the turntable speed. Figure 2 The three rotation speeds of slow, normal and fast are shown in FIG. This is only an example, and the embodiment of the present application does not limit the method of adjusting the rotation speed.

[0150] In an optional manner, in order to enrich the content of the music playing interface, the music client can make the first foreground image change according to the rhythm information during the music playing process, and the rhythm information is the rhythm information of the currently playing music.

[0151] Optionally, the rhythm information includes one or more of sound loudness, frequency information or timbre sensitivity information.

[0152] In the case where the rhythm information includes sound loudness, the sound loudness is the size of the sound, and the music client determines the maximum value or average value of the sound loudness in each time period of the currently played music, and in the corresponding relationship between the sound loudness range and the jitter amplitude, determines the sound loudness range to which the maximum value or the average value belongs, and determines the jitter amplitude corresponding to the sound loudness range. When the music client plays the music and reaches the corresponding time period, it displays the first foreground image according to the jitter amplitude.

[0153] In the case where the rhythm information includes frequency information, the music client obtains the audio data of the currently playing music, performs Fourier transform on the audio data, obtains frequency domain data, and samples N (N is greater than 1, such as N is 16, etc.) spectrum data sources from the frequency band that humans are sensitive to (such as the frequency band of 20Hz to 700Hz). Then analyze these N data sources, take out the one with the highest current amplitude, and then use the change value of the data source (the difference between the current value and the previous value, the previous value can be understood as the value at the previous time point), and then perform a normalization process of visual amplification, such as amplifying the data of 0.35 to 0.7 to 0 to 1. Then use the accumulated value of the change value instead of the time parameter for coloring, so that the faster the change value, the faster the change speed of the first foreground image distortion, so that the first foreground image changes with the sensitive rhythm of the music. Alternatively, use the change value as the amplitude of the screen particle fission effect, so that the first foreground image will vibrate with the change of the music.

[0154] The process of sampling N spectrum data sources from a frequency segment is as follows: the frequency segment is equally divided into N-1 sub-frequency segments, so that N points of spectrum data sources can be sampled.

[0155] It should be noted that the reason for using the variation value instead of directly using the spectrum value is that the visual effect requires more intuitive and sensitive feedback, but the spectrum value will fluctuate within a stable range, and the variation value is more exaggerated than the spectrum value. Therefore, using the variation value here will have a better visual effect and a better user experience.

[0156] In the case where the rhythm information includes timbre sensitive information, the music client obtains audio data of the currently played music, and uses a timbre extraction model to extract the timbre data. In the timbre data, the timbre sensitive information of each playing time point is extracted, and the change of the first foreground image is controlled according to the timbre sensitive information.

[0157] Optionally, the change of the first foreground image may be understood as the presence of an animation effect.

[0158] In an optional manner, as the background of the music playback interface, the clarity of the image is preferably guaranteed. Considering that the images uploaded by users come from different sources and have different qualities, when the clarity of the image is low, the image can be super-resolution processed to improve the clarity of the image, so that the music playback interface has a better display effect. There are two opportunities to perform super-resolution processing on the image, which are explained below.

[0159] The first type: After step 104, the UI layer of the player-style setting interface displays a super-resolution option. If the user thinks that the target image is not clear enough, the user can click on the super-resolution option, and the music client sends a first request to the server. The first request is used to instruct the super-resolution and foreground and background separation processing of the target image. The first request includes the target image. The server performs super-resolution processing on the target image to improve the clarity of the target image to obtain a super-resolution image. The server separates the background and foreground of the super-resolution image to obtain the foreground layer information of the super-resolution image, which is called the second foreground layer information. The server sends the second foreground layer information and the super-resolution image to the music client. The music client determines the third foreground image based on the second foreground layer information and the super-resolution image. The music client displays the super-resolution image on the background layer and the third foreground image on the foreground layer.

[0160] In addition, if the user believes that the clarity of the target image is sufficient, the user can click the use option or the completion option to make the music client jump to the music playback interface, which is the music playback interface currently set by the user.

[0161] Optionally, during the super-resolution process, the user can also operate the target image. Operation options are displayed in the player's style setting interface. The user can click on the operation options, and the music client receives the operation instructions and performs the operation indicated by the operation instructions on the target image, wherein the operation instructions include one or more of move instructions, crop instructions or edit instructions. Move refers to moving the position of the target image, cropping refers to cropping the target image, and editing includes but is not limited to adjusting the color or style of the target image. After the music client determines the third foreground image, the same operation as the operation is performed on the super-resolution image and the third foreground image. In this way, the operation on the target image also corresponds to the super-resolution image and the third foreground image, and the user does not need to perform the operation twice. For example, after the target image is cropped, the corresponding super-resolution image is also cropped.

[0162] The second method is to select super-resolution processing for the target image when the user first uploads the target image. The process is:

[0163] After the user uploads the target image through the player-style setting interface, a super-resolution option is provided in the setting interface, and the user can click on the super-resolution option. In response to receiving the trigger instruction of the super-resolution option, the music client sends a second request to the server, and the second request is used to instruct the super-resolution and foreground and background separation processing of the target image. The second request includes the target image. After receiving the second request, the server obtains the target image from it. Before super-resolution processing of the target image, the server performs super-resolution processing on the target object to obtain a super-resolution image. The server separates the background and foreground of the super-resolution image to obtain the foreground layer information of the super-resolution image, which is called the second foreground layer information. The server sends the second foreground layer information and the super-resolution image to the music client. The music client determines the third foreground image based on the second foreground layer information and the super-resolution image. The music client displays the super-resolution image on the background layer and the third foreground image on the foreground layer.

[0164] Here, the separation process refers to the process of step 102, and the process of determining the third foreground image refers to the process of step 103, which will not be repeated here.

[0165] When the third foreground image is subsequently displayed, the third foreground image can also change according to the rhythm information, and the relationship between the layers can be adjusted, and the super-resolved image can also be adjusted. For these processes, please refer to the description of the first foreground image in the previous article and will not be repeated here.

[0166] Optionally, during the super-resolution process, the music client can display whether the currently displayed image is the original image or the super-resolution image. Figure 7 As shown in the figure, the original image quality corresponds to the original image, and the enhanced image quality corresponds to the super-resolution image. Before enhancement, the original image quality is displayed, during enhancement, the enhanced image quality is displayed, and after enhancement, the user can choose to display the original image or the super-resolution image. If the user chooses the original image quality, the original image is displayed, and if the user chooses the enhanced image quality, the super-resolution image is displayed. In this way, the user can understand the super-resolution progress in a timely manner.

[0167] Optionally, the server side may not send the second foreground layer information to the music server side, and use the first foreground layer information when determining the third foreground image. However, since a higher-definition image is used to separate the foreground and background, the separation result can be more accurate.

[0168] Optionally, the super-resolution process is:

[0169] The server uses the super-resolution model to perform super-resolution processing on the target image to obtain a super-resolution image. Figure 8As shown, the super-resolution model includes an image encoder, a Diffusion denoising model, an adaptive enhancement model and an image decoder. Before the adaptive enhancement model is used for processing, the image is also subjected to a prediction process of blur and / or noise level. Among them, the image encoder includes multiple convolutional layers, the adaptive enhancement model is an attention model including convolutional layers, and the Diffusion denoising model can adopt a Diffusion denoising model of any architecture, such as the Diffusion denoising model is a U-NetDiffusion denoising model. The decoding process of the image decoder is inverse to the encoding process of the image encoder.

[0170] First, the training process of the super-resolution model is described. The training principle of the image encoder is: low-quality images have a different distribution than normal images. Using the encoder of normal images will mask the underlying features and thus lose the true expression of the low-quality image. Therefore, it is necessary to fine-tune the parameters of the encoder for low-quality images.

[0171] The training principle of the adaptive enhancement model is: there are various types of image degradation in the real world. Using a unified image enhancement module will not adequately enhance stronger degradation, resulting in the actual result still not meeting the clarity requirements, or excessively enhancing weaker degradation, resulting in bad cases. Therefore, combining the adaptive enhancement model can cope with various user images and achieve a clear effect after enhancement.

[0172] The training principle of the Diffusion denoising model is: the traditional image enhancement model can repair the image under certain image loss, but it does not improve much for images with more loss. The reason is that it has no generation ability and cannot imagine the original appearance of the lost part. The Diffusion denoising method has natural generation ability. Based on big data training, it can achieve good generation effects for various scenarios and is very suitable for supplementing the lost part of low-quality images. Therefore, the embodiment of the present application adopts the Diffusion denoising model as the basis of the generative ability. The weight of this part will also be updated, but it is updated through LORA (low-rank adaptation), which is more convenient.

[0173] The super-resolution model adopts an end-to-end design to obtain high-definition images from the real world, and uses blurring, noise addition, and image compression to construct low-quality images to simulate low-quality images from the real world. The low-quality images are input into the super-resolution model for training to obtain predicted images. Then, the parameters of the image encoder and the adaptive enhancement model are updated through back propagation, so that the predicted image fits the high-quality image paired with the low-quality image as much as possible. Pairing means that when a low-quality image is generated from a high-quality image, the low-quality image is paired with the high-quality image.

[0174] Here, the device for training the super-resolution model can be a server with a server-side installed, or it can be other servers. After the training is completed on other servers, it is deployed to the server with the server-side installed.

[0175] The server performs image encoding processing on the target image to obtain an encoding result, which is a multidimensional vector. The target image is also predicted for blur and / or noise to obtain a prediction result, which is used to indicate the degree of image degradation. The server performs adaptive enhancement processing based on the prediction result to obtain an enhancement result, so that enhancement can be performed according to the degree of degradation, and the enhancement effect is good. The server inputs the encoding result and the enhancement result into the Diffusion denoising model to obtain a denoising result. The server performs image decoding processing on the denoising result to restore the image, that is, to obtain a super-resolution image.

[0176] When the face or text is very small, in order to make the super-resolution image better, the server performs the following processing:

[0177] The server decodes the denoising result to obtain an intermediate image, and detects the face and / or text in the intermediate image to obtain the position of the face in the image and the position of the text in the image. The server inputs the target image into the GAN (generative adversarial networks) model to obtain the output result of the GAN model. For the detected face and text parts, the server fuses the detection result and the output result to obtain the fused face and text parts, and uses the decoded image for the other parts, so that the super-resolution image can be obtained.

[0178] The fusion processing here can be averaging the pixel values, or only using the pixel values ​​of the face and text in the output result, or weighting the pixel values ​​of the face and text in the detection result and the output result.

[0179] In an embodiment of the present application, a U-Net trained to perform foreground and background separation is also provided, and the training process is as follows.

[0180] First, use a segmentation model (such as a deep learning convolutional neural network model) to process the input image and identify the main part of the image, which is the foreground data we need. This process may involve different tasks such as portrait segmentation and object segmentation. Then, the segmentation mask is manually annotated to filter out the foreground area that is considered to meet the requirements. The pixel values ​​included in the segmentation mask are 0 or 255, 0 corresponds to complete transparency, and 255 corresponds to complete opacity. However, in order to make the edges of the foreground area smoother, the segmentation mask is then converted to a matting mask. However, it is usually difficult to manually annotate the matting mask. Here, the segmentation mask is input into the Matting Anything pre-training model to obtain the matting mask. In this way, the image and the corresponding matting mask are obtained, that is, the training sample is obtained.

[0181] Then, the initial architecture of U-Net is trained using the training samples so that the output of U-Net is close to the matting mask in the training samples to obtain the final U-Net.

[0182] Here, the device for training U-Net can be a server with the server side installed, or it can be other servers. After the training is completed on other servers, it is deployed to the server with the server side installed.

[0183] The embodiment of the present application also provides a display system for a music playback interface, which includes a terminal and a server, wherein a music client is installed on the terminal and a server is installed on the server.

[0184] It should be noted that in the attached Figures 5 to 7 In order to facilitate understanding, the foreground and background are marked. In actual display, the words foreground and background will not be displayed in the interface.

[0185] Fig. 9 The structure block diagram of a terminal 900 provided by an exemplary embodiment of the present application is shown. The terminal 900 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), an MP4 player (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4), a laptop computer or a desktop computer. The terminal 900 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0186] Typically, the terminal 900 includes a processor 901 and a memory 902 .

[0187] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0188] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one program instruction, which is used to be executed by the processor 901 to implement the display method of the music playback interface provided in the method embodiment of the present application.

[0189] In some embodiments, the terminal 900 may further optionally include: a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902 and the peripheral device interface 903 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 903 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 904, a touch display screen 905, a camera 906, an audio circuit 907, a positioning component 908 and a power supply 909.

[0190] The peripheral device interface 903 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0191] The radio frequency circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 904 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 904 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 904 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0192] The display screen 905 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 905 is a touch display screen, the display screen 905 also has the ability to collect touch signals on the surface or above the surface of the display screen 905. The touch signal can be input to the processor 901 as a control signal for processing. At this time, the display screen 905 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 905 can be one, and the front panel of the terminal 900 is set; in other embodiments, the display screen 905 can be at least two, which are respectively set on different surfaces of the terminal 900 or are folded; in some other embodiments, the display screen 905 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 900. Even, the display screen 905 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 905 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0193] The camera assembly 906 is used to capture images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0194] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 901 for processing, or input them into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 900. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0195] The positioning component 908 is used to locate the current geographical location of the terminal 900 to implement navigation or LBS (Location Based Service). The positioning component 908 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system or the European Union's Galileo system.

[0196] The power supply 909 is used to power various components in the terminal 900. The power supply 909 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 909 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0197] In some embodiments, the terminal 900 further includes one or more sensors 910 , including but not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , a fingerprint sensor 914 , an optical sensor 915 , and a proximity sensor 916 .

[0198] The acceleration sensor 911 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 900. For example, the acceleration sensor 911 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 901 can control the touch display screen 905 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 911. The acceleration sensor 911 can also be used for collecting game or user motion data.

[0199] The gyro sensor 912 can detect the body direction and rotation angle of the terminal 900, and the gyro sensor 912 can cooperate with the acceleration sensor 911 to collect the user's 3D actions on the terminal 900. The processor 901 can implement the following functions based on the data collected by the gyro sensor 912: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0200] The pressure sensor 913 can be set on the side frame of the terminal 900 and / or the lower layer of the touch display screen 905. When the pressure sensor 913 is set on the side frame of the terminal 900, it can detect the user's holding signal of the terminal 900, and the processor 901 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 913. When the pressure sensor 913 is set on the lower layer of the touch display screen 905, the processor 901 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 905. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0201] The fingerprint sensor 914 is used to collect the user's fingerprint, and the processor 901 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 914, or the fingerprint sensor 914 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 901 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 914 can be set on the front, back, or side of the terminal 900. When a physical button or a manufacturer logo is set on the terminal 900, the fingerprint sensor 914 can be integrated with the physical button or the manufacturer logo.

[0202] The optical sensor 915 is used to collect the ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the touch display screen 905 according to the ambient light intensity collected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 905 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 905 is reduced. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera component 906 according to the ambient light intensity collected by the optical sensor 915.

[0203] The proximity sensor 916, also called a distance sensor, is usually arranged on the front panel of the terminal 900. The proximity sensor 916 is used to collect the distance between the user and the front of the terminal 900. In one embodiment, when the proximity sensor 916 detects that the distance between the user and the front of the terminal 900 is gradually decreasing, the processor 901 controls the touch display screen 905 to switch from the screen-on state to the screen-off state; when the proximity sensor 916 detects that the distance between the user and the front of the terminal 900 is gradually increasing, the processor 901 controls the touch display screen 905 to switch from the screen-off state to the screen-on state.

[0204] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the terminal 900, and the terminal 900 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0205] Fig.10 It is a structural diagram of a server provided in an embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1001 and one or more memories 1002, wherein the memory 1002 stores at least one program instruction, and the at least one program instruction is loaded and executed by the processor 1001 to implement the display method of the music playback interface provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be described in detail here.

[0206] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including program instructions, and the program instructions can be executed by a processor in a terminal and a server to complete the display method of the music playback interface in the above embodiment. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device.

[0207] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0208] In the embodiment of the present application, "A and / or B" includes three situations, namely A, B and A and B.

[0209] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for displaying a music playing interface, applied to a music client, characterized in that: The music playing interface includes a foreground layer, a background layer and a song related layer, and the method includes: The music client provides a player-style setting interface for sending the target image uploaded by the user to the server; The server performs separation processing on the background and foreground of the target image to obtain first foreground layer information of the target image, and sends the first foreground layer information to the music client; The music client determines a first foreground image based on the first foreground layer information and the target image; The music client displays the target image on the background layer, displays the first foreground image on the foreground layer, and displays the song-related layer between the background layer and the foreground layer.

2. The method according to claim 1, characterized in that The step of displaying the first foreground image on the foreground layer includes: The music client displays the first foreground image in a target area in the foreground layer, or displays a non-completely transparent portion of the first foreground image in the target area, wherein the target area covers a foreground area corresponding to the target image in the background layer.

3. The method according to claim 1 or 2, characterized in that: The first foreground layer information includes mask information; The music client determines the first foreground image based on the first foreground layer information and the target image, including: The music client performs mixing processing on the target image using the mask information to obtain the first foreground image.

4. The method according to claim 3, characterized in that The music client performs a mixing process on the target image using the mask information to obtain the first foreground image, including: The music client determines the alpha value of the pixel at the corresponding position in the target image using the value of each position in the mask information, and combines the red, green, and blue (RGB) values ​​of each pixel in the target image with the alpha value to obtain the pixel value of each pixel in the first foreground image; or The music client uses the numerical value of each position in the mask information to determine the alpha value of the corresponding pixel in the target image, combines the RGB value of each pixel in the target image with the alpha value to obtain the pixel value of each pixel in the second foreground image, and uses the pixel value of each pixel in the second foreground image to amplify the second foreground image to obtain the first foreground image.

5. The method according to any one of claims 1 to 4, characterized in that: The server performs separation processing on the background and foreground of the target image to obtain first foreground layer information of the target image, including: The server performs target type recognition on the target image to obtain a type recognition result; the server performs separation processing on the target image based on the type recognition result to obtain first foreground layer information containing content of the target type; or The server inputs the target image into a U-shaped network U-Net to obtain first foreground layer information containing content of the target type, and the U-Net is used to identify the content of the target type; The target type includes a person type and / or an animal type.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: When the proportion of the occluded area to the area of ​​the song-related layer exceeds the target threshold, the music client displays the foreground layer between the song-related layer and the background layer, wherein the occluded area is the area of ​​the song-related layer obscured by the foreground layer.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The music client selects one of the foreground layer, the background layer and the song-related layer as a target layer in response to the layer selection instruction; The music client adjusts the target layer in response to the instruction to adjust the target layer.

8. The method according to claim 7, characterized in that The adjustment process includes adjusting the position and size of the image within the layer or adjusting the position between layers.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: In response to receiving the adjustment instruction of the target image, the music client performs a first operation indicated by the adjustment instruction on the target image in the background layer; The adjustment instruction includes an instruction for adjusting size, an instruction for adjusting position or an instruction for adjusting style.

10. The method according to claim 9, characterized in that The method further comprises: The music client determines, in response to performing the first operation on the target image, a second operation to be performed on the first foreground image, wherein the second operation corresponds to the first operation; The second operation is performed on the first foreground image in the foreground layer.

11. The method according to any one of claims 1 to 10, characterized in that: The song-related layer includes a record player icon and / or a song icon; The method further comprises: The music client controls the rotation of the record player icon and / or song icon in the song-related layer.

12. The method according to any one of claims 1 to 11, characterized in that: The music playing interface also includes a user interface layer; The method further comprises: The music client displays the user interface layer on the top layer of the music playing interface.

13. The method according to any one of claims 1 to 12, characterized in that: The method further comprises: The music client drives the first foreground image in the foreground layer to change according to the rhythm information, wherein the rhythm information is the rhythm information of the music currently played by the music client.

14. The method according to claim 13, characterized in that The rhythm information includes one or more of sound loudness, frequency information or timbre sensitivity information.

15. The method according to any one of claims 1 to 14, characterized in that After displaying the first foreground image on the foreground layer, the method comprises: The music client sends a first request to the server, wherein the first request includes the target image; The server performs super-resolution processing on the target image to obtain a super-resolution image, and separates the background and foreground of the super-resolution image to obtain second foreground layer information of the super-resolution image; The server sends the second foreground layer information and the super-resolved image to the music client; The music client determines a third foreground image based on the second foreground layer information and the super-resolved image; The music client displays the super-resolved image on the background layer, and displays the third foreground image on the foreground layer.

16. The method according to claim 15, characterized in that The music client displays the super-resolved image on the background layer and displays the third foreground image before the foreground layer, and the method further includes: In response to receiving the operation instruction for the target image, the music client performs the operation indicated by the operation instruction on the target image, wherein the operation instruction includes one or more of a move instruction, a crop instruction or an edit instruction; The music client performs an operation corresponding to the operation on the super-resolved image and the third foreground image.

17. The method according to any one of claims 1 to 16, characterized in that The music client provides a player-style setting interface for sending the target image uploaded by the user to the server, including: The music client provides a player-style setting interface for providing super-resolution options after receiving a target image uploaded by a user; In response to receiving the trigger instruction of the super-resolution option, the music client sends a second request to the server, wherein the second request includes the target image; Before the server performs separation processing on the background and foreground of the target image, the method further includes: the server performs super-resolution processing on the target image; After the server performs separation processing on the background and foreground of the target image, the method further includes: sending the super-resolved image to the music client; The music client uses the super-resolved image as the target image.

18. A music playing interface display system, characterized in that: The display system includes a terminal and a server; The terminal is installed with a music client according to any one of claims 1 to 17; The server is installed with the server side described in any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that: The storage medium stores at least one program instruction, which is loaded and executed by the processor to implement the operations performed by the method for displaying a music playback interface as described in any one of claims 1 to 17.

Images