Display method, medium, and electronic device

Abstract

The present application relates to the technical field of computers, in particular to a display method, a medium, and an electronic device. In the display method, when a specific relative angle and a specific relative distance exist between a first electronic device and a second electronic device, after a screen of the first electronic device displays a captured picture showing a specific placement position where a user wants a graphic element to be placed on a screen of the second electronic device, the screen of the second electronic device, at such time, displays the graphic element received from the first electronic device, and displays the graphic element according to the corresponding display angle and display size. It can be understood that when the second electronic device performs display according to the corresponding display angle and display size, the manner in which the graphic element displayed on the second electronic device is placed matches the angle of rotation and relative size of the graphic element of the captured picture on the first electronic device relative to a screen region of the second electronic device, thereby improving the interactivity of the user.

Description

Display method, medium, and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 3, 2023, with application number 202310979960.6 and application name “Display Method, Medium and Electronic Device”. The entire contents of the above application are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a display method, a medium, and an electronic device. Background Art

[0003] Currently, data can be shared between various electronic devices. For example, devices can share images, text, and other data through common methods such as Bluetooth and wireless networks. When sharing images between electronic devices, users can use their devices to share the entire image or just the image elements (for example, specific elements such as people and objects in the image).

[0004] For example, Figure 1 shows a schematic diagram of a mobile phone 100 using a cloud 300 to share data with a computer 200, wherein the mobile phone 100 can transmit data to the computer 200 via the cloud 300. As shown in Figure 1, a user is creating a poster using image processing software (Photoshop) on a computer and wants to obtain some image material. However, it is not convenient for the user to use computer 200 to capture the material, so the user needs to use mobile phone 100 to obtain the required image material and transfer it to computer 200. Sometimes, the image material contains a lot of information, and the user wants to directly obtain specific image elements from the image material without the cluttered background information. Mobile phone 100 can extract the image elements from the image material and transfer them to computer 200, allowing computer 200 to process the image elements.

[0005] Specifically, a user can use a mobile phone 100 to obtain an image 1 as shown in FIG1 . At this time, the mobile phone 100 can send the image 1 to the cloud 300. The intelligent cutout technology in the cloud 300 can extract the various primitives in the image 1. For example, the "apple" primitive C1 in the image 1 can be extracted and then sent to the mobile phone 100. The extracted primitives, such as the "apple" primitive, will be displayed on the mobile phone 100. Since multiple primitives may be extracted sometimes, the user can select the primitive to be transmitted from the multiple primitives. For example, the user selects the "apple" primitive. At this time, the mobile phone 100 will send the "apple" primitive to the cloud 300, which will then forward it to the computer 200, so that the computer 200 will display it at a fixed position on the image processing software interface.

[0006] However, since the computer 200 can only display the received graphic elements in a fixed form and at a fixed position, the user cannot directly obtain the desired placement effect, and needs to adjust the placement position of the graphic elements on the screen through other operations such as dragging, which has poor interactivity.

[0007] Summary of the Invention

[0008] In order to solve the above problems, the present application provides a display method, a medium and an electronic device. In the display method, when there is a specific relative angle and a specific relative distance between the first electronic device and the second electronic device, after the screen of the first electronic device displays a photographic picture of the specific placement position of the graphic element that the user wants on the screen of the second electronic device, the screen of the second electronic device displays the graphic element received from the first electronic device, and the graphic element is displayed according to the corresponding display angle and display size. It can be understood that when the second electronic device is displayed according to the corresponding display angle and display size, the placement of the graphic element displayed on the second electronic device achieves an effect that is consistent with the rotation angle and relative size of the graphic element on the photographic picture on the first electronic device relative to the screen area of ​​the second electronic device, thereby improving user interactivity.

[0009] In a first aspect, an embodiment of the present application provides a display method, including: a first electronic device detects a user's first sharing operation of sharing first information with a second electronic device, and the first electronic device photographs the display screen of the second electronic device with first position information; the second electronic device displays the first information with first display parameters; the first electronic device detects a user's second sharing operation of sharing second information with the second electronic device, and the first electronic device photographs the display screen of the second electronic device with second position information; the second electronic device displays the second information with second display parameters; wherein the first position information and the second position information include at least one of a relative distance and a relative angle between the first electronic device and the second electronic device, and the first display parameter and the second display parameter include at least one of a display ratio and a display angle.

[0010] It is understandable that the user's sharing operation can be in various forms. For example, it can be a physical operation such as a user clicking on the screen of the first electronic device, or the user is currently still facing the screen of the second electronic device for a preset time.

[0011] It is understandable that the first electronic device detects the user's first sharing operation, and the first electronic device captures the display screen of the second electronic device corresponding to the first position information. The captured picture at this time includes the specific placement position of the first information relative to the captured second electronic device, such as the rotation angle and relative size of the first information relative to the screen of the second electronic device. It is understandable that the display ratio and display angle in the first display parameter correspond to the captured picture in the first electronic device. At this time, when the second electronic device displays the first information with the corresponding first display parameter, since the first display parameter includes at least one of the display ratio and display angle, when the second electronic device displays the first information, the first information will produce the effect of the rotation angle and relative size relative to the second electronic device when displayed on the first electronic device on the screen of the second electronic device, thereby improving user interactivity.

[0012] In a possible implementation of the first aspect, the first information and the second information are images or graphic elements in an image.

[0013] In a possible implementation of the first aspect, the display ratio is a ratio between a size of the first information displayed on the second electronic device and a size of the first information displayed on the first electronic device.

[0014] In a possible implementation of the first aspect above, the first position information includes a first relative distance, and the first display parameter includes a first display ratio; the second position information includes a second relative distance, and the second display parameter includes a second display ratio, wherein the first relative distance is smaller than the second relative distance, and the first display ratio is smaller than the second display ratio.

[0015] It is understandable that when a user uses the screen of a first electronic device to shoot a picture of a second electronic device, the user can adjust the relative size of the first information displayed on the screen and the screen of the second electronic device by changing the relative distance accordingly by zooming in / out on the first electronic device. When the screen of the first electronic device is away from the screen of the second electronic device, the screen of the second electronic device in the picture displayed on the screen of the first electronic device will become smaller, and since the size of the first information remains unchanged, the first information will become larger relative to the screen of the second electronic device, so the display ratio is relatively large; for another example, when the user brings the screen of the first electronic device close to the screen of the second electronic device, the screen of the second electronic device in the picture displayed on the screen of the first electronic device will become larger, and since the size of the first information remains unchanged, the first information will become smaller relative to the screen of the second electronic device, so the display ratio is relatively small.

[0016] It is understandable that by adjusting the relative distance between the first electronic device and the second electronic device, the user can adjust the size of the graphic element / image displayed on the second electronic device accordingly, thereby improving the user experience.

[0017] In a possible implementation of the first aspect above, it also includes: the first position information includes a first relative angle, and the first display parameter includes the first display angle; the second position information includes a second relative angle, and the second display parameter includes the second display angle, wherein the first relative angle is smaller than the second relative angle, and the first display angle is smaller than the second display angle.

[0018] It is understandable that when a user uses the screen of a first electronic device to shoot the screen of a second electronic device, the user can adjust the rotation angle of the first information displayed on the screen of the first electronic device relative to the screen of the second electronic device by changing the relative angle between the first electronic device and the second electronic device. It is understandable that the display angle of the first information on the second electronic device is proportional to the rotation angle of the first information displayed on the screen of the first electronic device relative to the screen of the second electronic device. Since the rotation angle of the first information displayed on the screen of the first electronic device relative to the screen of the second electronic device is proportional to the relative angle between the first electronic device and the second electronic device. Therefore, the larger the first relative angle, the larger the corresponding first display angle.

[0019] It is understandable that by adjusting the relative angle between the first electronic device and the second electronic device, the user can adjust the display angle of the graphic element / image on the second electronic device accordingly, thereby improving the user experience.

[0020] In a possible implementation of the first aspect above, the first information includes a first graphic element in the first image, and the method also includes: after the first electronic device detects the first sharing operation, it extracts the first image or multiple graphic elements in the first area selected by the user in the first image; the first electronic device displays multiple graphic elements and sends the first graphic element selected by the user from the multiple graphic elements to the second electronic device.

[0021] It can be understood that after the first electronic device extracts multiple graphic elements, the user can select the graphic elements that he or she desires to transmit, which provides the user with many options and improves the user's selectivity.

[0022] In a possible implementation of the first aspect above, the first information includes a first graphic element in the first image, and the method also includes: the first electronic device extracts a graphic element in the first image after detecting the first sharing operation; the first electronic device displays the extracted graphic element as the first graphic element and sends it to the second electronic device.

[0023] It can be understood that after the first electronic device extracts a graphic element, it transmits it to the second electronic device, thereby improving the efficiency of the sharing operation.

[0024] In a possible implementation of the first aspect above, the first information is displayed in a first application interface of the second electronic device.

[0025] In a possible implementation of the first aspect above, it also includes: the first electronic device sends to the second electronic device first screen background image information of the first screen background image (i.e., the first screen background hereinafter) obtained by photographing the display screen of the second electronic device using the first location information; and the second electronic device determines the first display parameter based on the first screen background image information.

[0026] In a possible implementation of the first aspect above, the first screen background image information includes multiple first image feature points of the first screen background image (for example, the first feature points hereinafter); and the second electronic device determines the first display parameters based on the first screen background image information, including: the second electronic device determines the first display parameters based on the multiple first image feature points of the first screen background image, and multiple second image feature points (for example, the second feature points hereinafter) corresponding to the multiple first image feature points in the first screen image of the current display interface of the second electronic device.

[0027] It can be understood that the first screen image of the current display interface of the second electronic device is the second screen background below.

[0028] It can be understood that the first electronic device sends the image feature points as screen background image information to the second electronic device. Since the image feature points are smaller than the original image data, transmitting the feature point information can effectively save the amount of transmitted data, thereby reducing transmission power consumption.

[0029] In a possible implementation of the first aspect above, the first display parameter includes a first display ratio, and the second electronic device determines the first display parameter based on multiple first image feature points of the first screen background image and multiple second image feature points corresponding to the multiple first image feature points in the first screen image of the current display interface of the second electronic device, including: obtaining a spatial transformation matrix based on the multiple first image feature points and the multiple second image feature points; determining, based on the spatial transformation matrix, a second center point and a second offset point corresponding to the first center point and the first offset point of the first screen background image from the first screen image of the current display interface of the second electronic device; calculating a first distance based on the coordinates of the first center point and the coordinates of the first offset point, and calculating a second distance based on the coordinates of the second center point and the coordinates of the second offset point; and obtaining a first display ratio based on the ratio between the first distance and the second distance.

[0030] In a possible implementation of the first aspect above, the first display parameter includes a first display angle, and the second electronic device determines the first display parameter based on multiple first image feature points of the first screen background image and multiple second image feature points corresponding to the multiple first image feature points in the first screen image of the current display interface of the second electronic device, including: obtaining a spatial transformation matrix based on the multiple first image feature points and the multiple second image feature points; determining, based on the spatial transformation matrix, a second center point and a second offset point corresponding to the first center point and the first offset point of the first screen background image from the first screen image of the current display interface of the second electronic device; calculating a first angle between a straight line including the first center point and the first offset point and a first coordinate axis in the coordinate system of the second electronic device, and calculating a second angle between a straight line including the second center point and the second offset point and the first coordinate axis in the coordinate system of the second electronic device; and obtaining the first display angle based on the difference between the first angle and the second angle.

[0031] In a second aspect, an embodiment of the present application provides an electronic device comprising: one or more processors; one or more memories; one or more memories storing one or more programs, wherein when one or more programs are executed by one or more processors, the electronic device executes any one of the display methods described in the first aspect and various implementations of the first aspect.

[0032] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the computer executes the display method as described in the first aspect and any one of the various implementations of the first aspect.

[0033] In a fourth aspect, a computer program product is provided, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, the processor in the electronic device executes a display method as described in the first aspect or any one of the possible implementations of the first aspect.

[0034] In a fifth aspect, a chip is provided. When the chip is running, the chip executes the display method of the first aspect or any one of the possible implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] FIG1 shows a schematic diagram of a mobile phone 100 sharing data with a computer 200 using a cloud 300 according to some embodiments of the present application;

[0036] FIG2 is a schematic diagram showing a process in which a mobile phone 100 extracts graphic elements through a computer 200 and displays the graphic elements at corresponding positions on the computer 200 according to some embodiments of the present application;

[0037] FIG3A shows a schematic diagram of a connection between a mobile phone 100 and a computer 200 according to some embodiments of the present application;

[0038] FIG3B shows a schematic diagram of a process of extracting and displaying graphic primitives by the mobile phone 100 according to some embodiments of the present application;

[0039] FIG3C is a schematic diagram showing a computer 200 displaying graphic elements after the mobile phone 100 is rotated according to some embodiments of the present application;

[0040] FIG4A is a schematic diagram showing another embodiment of the present application, in which the computer 200 displays graphic elements after the mobile phone 100 is rotated;

[0041] FIG4B shows a schematic diagram of a first screen background B1 according to some embodiments of the present application;

[0042] FIG4C shows a schematic diagram of mobile phone screens W4 and W5 corresponding to a distance h1 between the mobile phone 100 and the computer 200 screen, and a distance h2 between the mobile phone 100 and the computer 200 screen, when the mobile phone 100 and the computer 200 are at the same relative angle, according to some embodiments of the present application;

[0043] FIG5 is a schematic diagram showing a display method process according to some embodiments of the present application;

[0044] FIG6A shows a schematic diagram of a mobile phone 100 capturing an image by shooting according to some embodiments of the present application;

[0045] FIG6B shows a schematic diagram of a mobile phone 100 acquiring a foreground image through a gallery according to some embodiments of the present application;

[0046] FIG6C is a schematic diagram showing a method of transmitting displayed graphic elements from the mobile phone 100 to the computer 200 according to some embodiments of the present application;

[0047] FIG6D shows a perspective schematic diagram of a computer 200 displayed on the screen of a mobile phone 100 according to some embodiments of the present application;

[0048] FIG6E shows a schematic diagram of a correspondence between pixel points on a first screen background of a mobile phone 100 and pixel points on a second screen background of a computer 200 according to some embodiments of the present application;

[0049] FIG6F is a schematic diagram showing a comparison between a display of the computer 200 without and with post-processing optimization according to some embodiments of the present application;

[0050] FIG7A is a schematic diagram showing a process of obtaining a display angle and a display ratio according to some embodiments of the present application;

[0051] FIG7B shows a schematic diagram illustrating correspondence between center points and offset points in a first screen background and a second screen background according to some embodiments of the present application;

[0052] FIG8 shows a schematic diagram of the functional module structure of a mobile phone 100 according to some embodiments of the present application;

[0053] FIG9 shows a schematic diagram of a software system structure of a mobile phone 100 according to some embodiments of the present application;

[0054] FIG10 shows a schematic diagram of the hardware structure of an electronic device according to some embodiments of the present application. DETAILED DESCRIPTION

[0055] The illustrative embodiments of the present application include, but are not limited to, a display method, a medium, and an electronic device.

[0056] The technical solution of this application is described in detail below with reference to the accompanying drawings.

[0057] As described in the background art, since the computer 200 can only display graphics elements at fixed positions on the screen, it cannot directly display the graphics elements at the position desired by the user, resulting in poor interactivity with the user.

[0058] In some embodiments, after the mobile phone 100 obtains the desired graphic element, the user can use the camera function of the mobile phone 100 to take a picture of the screen of the computer 200 while the screen is displaying the graphic element. At this time, the screen of the mobile phone 100 can show the specific location of the graphic element on the computer 200. The mobile phone 100 then sends the image showing the specific location of the graphic element on the screen of the computer 200 to the computer 200, and the computer 200 can then place the graphic element at the corresponding location.

[0059] Specifically, FIG2 illustrates a schematic diagram of the process by which the mobile phone 100 extracts graphic elements through the computer 200 and displays the graphic elements at corresponding locations on the computer 200. Furthermore, the mobile phone 100 and the computer 200 establish a communication connection via wireless communication technology, such as Bluetooth or a wireless network, for data transmission. For example, as shown in FIG2 , the process may include the following steps:

[0060] S01: The mobile phone 100 sends the acquired foreground image to the computer 200, where the foreground image includes the image elements to be extracted.

[0061] It can be understood that an image includes multiple regions with significant features. For example, the regions with significant features (ie, significant regions) may be specific people, objects, text, etc., and the regions with significant features may serve as image primitives.

[0062] For example, as shown in FIG3A , a user uses the image processing software on the computer 200 to create a poster. At this time, there is already a "basket" on the poster. The user can use the material sharing software on the mobile phone 100 to obtain image materials. The computer 200 and the mobile phone 100 are connected through the network, and the material sharing software on the mobile phone 100 supports the transmission of image elements with the image processing software on the computer 200. The "shoot" button P1 in the material sharing software on the mobile phone 100 supports shooting and obtaining the foreground image, and the "gallery" button P2 in the material sharing software on the mobile phone 100 supports obtaining the foreground image from the gallery. After the user obtains the foreground image shown on the mobile phone screen W1 in FIG3B , he can click on the screen of the mobile phone 100. At this time, the mobile phone 100 will send the foreground image shown on the mobile phone screen W1 in FIG3B to the computer 200.

[0063] It is understandable that in addition to using the above-mentioned image processing software, the computer 200 and the mobile phone 100 can use the above-mentioned material sharing software as well as any other application, such as a third-party application or an application supported by the system, which is not limited here.

[0064] S02: The server in the computer 200 extracts the image primitives from the foreground image. For example, the image primitives in the foreground image may be extracted to obtain an "apple" image primitive C1 and a "bowl" image primitive C2.

[0065] S03: The computer 200 sends the extracted graphic elements to the mobile phone 100. For example, the computer 200 sends the extracted graphic element “apple” C1 and the graphic element “bowl” C2 to the mobile phone 100.

[0066] S04: After receiving the graphic element, the mobile phone 100 may select a graphic element with a larger size by default and display it on the screen.

[0067] For example, in Figure 3B , the "apple" icon C1 is displayed on the phone screen W2. While the icon is displayed, the phone 100 can capture the screen of the computer 200. When the phone 100 detects a user click, or automatically detects that the phone is pointing at the computer 200, the captured background image is obtained. For example, a user uses the phone's camera function to hold the phone 100, while displaying the "apple" icon, toward the capture area sq1 of the computer 200, resulting in the background image shown on the phone screen W3 in Figure 3B .

[0068] S05: The mobile phone 100 sends the background image to the computer 200.

[0069] S06: The server in the computer 200 calculates the center position of the screen of the computer 200 where the graphic element needs to be placed based on the background image. For example, based on the specific position of the center point Q1 of the "apple" graphic element C1 in the background image on the captured shooting area sq1, the server calculates the center position of the screen of the computer 200 where the center point Q1 of the "apple" graphic element C1 needs to be placed.

[0070] S07: The image processing software of the computer 200 calls a software interface to place the graphic element at the calculated center position. For example, in the display screen 002A of the computer 200 in FIG3B , the “apple” graphic element C1 is placed at the center position on the computer screen.

[0071] However, after receiving the background image, the computer 200 only calculates the center position of the graphic element displayed on the computer screen. At this time, the size or rotation angle of the graphic element in the background image relative to the shooting area is not considered during display, and the user interaction is not strong.

[0072] Specifically, the computer display does not consider the size of the image element in the background image relative to the captured area. It's understandable that, because the camera of mobile phone 100 has a perspective function, the captured area captured by mobile phone 100 at a certain relative distance from computer 200 is different from the actual size of the captured area. Relative distance can be understood as the distance from the center of gravity of mobile phone 100 to the computer's plane, where the computer's plane is the plane where the computer 200 screen resides. For example, on mobile phone screen W3 in FIG3B , the captured area displayed on the background image of mobile phone 100 is different in size from the actual captured area sq1. Therefore, when the image element is displayed in the captured area sq1 of computer 200 at the same size as in the background image, it cannot achieve the desired effect in the background image. For example, in FIG3B , the "apple" image element displayed on screen 002A of computer 200 occupies a smaller area of ​​the captured area than the "apple" image element in the background image of mobile phone 100.

[0073] Furthermore, the rotation angle of the primitives relative to the captured area in the background image is not taken into account during display.

[0074] It can be understood that the image data captured by the mobile phone 100 is represented by the XOY coordinate system as shown in Figure 3C (for example, the long frame of the mobile phone is the Y axis and the short frame of the mobile phone is the X axis), while the computer 200 is represented by the X'OY' coordinate system as shown in Figure 3C (for example, the long frame of the computer is the X axis and the short frame of the computer is the Y axis).

[0075] Since the mobile phone 100 can be rotated by the user during the shooting process to adjust the specific position of the image elements on the screen of the mobile phone 100 on the screen of the computer 200 being photographed, there is a relative angle between the mobile phone 100 and the computer 200 at this time, wherein the relative angle can be the angle value between the central axis of the screen of the mobile phone 100 and the central axis of the screen of the computer 200, wherein, when the mobile phone 100 and the computer 200 are placed upright along the direction of gravity, the central axes of the two are parallel.

[0076] When the phone 100 and computer 200 rotate relative to each other in space, the relative angle between them is the angle between the central axis Z1 of the phone 100 and the central axis Z2 of the computer 200, as shown in Figure 3C . In the image captured by the phone 100, the image element remains unchanged in the XOY coordinate system, but rotates in the X'OY' coordinate system. In the image captured by the phone 100, the computer 200 changes in the XOY coordinate system, but does not rotate in the X'OY' coordinate system. Therefore, the image element in the image captured by the phone 100 is rotated relative to the computer 200 screen. For example, using the X'OY' coordinate system as a standard, as shown in FIG3C , when mobile phone 100 displays mobile screen W3 as shown in FIG3C , the user rotates mobile phone 100. Since the specific position of the "apple" primitive on the mobile phone screen does not change, the "apple" primitive rotates with the rotation of the mobile phone. At this time, the "apple" primitive rotates in the X'OY' coordinate system, while the screen of computer 200 in the captured area of ​​the mobile phone image does not rotate in the X'OY' coordinate system. As a result, mobile phone 100 displays mobile screen W4 as shown in FIG3C , and the "apple" primitive rotates relative to the captured computer 200. When the user clicks, a background image such as mobile screen W4 is displayed. As can be seen from the background image, the user intended the "apple" primitive to be rotated and displayed on the captured screen of computer 200. However, the "apple" primitive displayed in the captured area sq2 on the display screen 002B of computer 200 in FIG3C does not rotate.

[0077] Therefore, in order to solve the above problems, the present application proposes a display method. When there is a specific relative angle and a specific relative distance between a first electronic device (such as a mobile phone 100) and a second electronic device (such as a computer 200), the screen of the first electronic device displays a photographic picture of the specific placement position of the graphic element that the user wants on the screen of the second electronic device (such as a computer 200). At this time, the second electronic device can adjust the display angle and display ratio of the graphic element on the screen of the second electronic device according to the rotation angle of the graphic element in the photographic picture of the first electronic device relative to the screen of the second electronic device, and the relative size of the graphic element relative to the photographed screen of the second electronic device, so that the placement of the graphic element displayed on the second electronic device achieves the effect of being consistent with the rotation angle and relative size of the graphic element in the photographic picture of the first electronic device relative to the screen area of ​​the second electronic device, thereby improving user interactivity.

[0078] The display angle of the graphic element on the screen of the second electronic device is the value of the angle between the central axis passing through the center point of the graphic element on the screen of the second electronic device and the central axis of the screen of the second electronic device.

[0079] The display ratio of the graphic element placed on the screen of the second electronic device is the ratio between the size of the graphic element displayed on the second electronic device and the size of the graphic element displayed on the first electronic device.

[0080] For example, as shown in FIG4A , for the mobile phone screen W4 of the mobile phone 100, the "apple" graphic element on the display screen 002C of the computer 200 is displayed according to the display angle corresponding to the rotation angle in the mobile phone screen W4 of the mobile phone 100, and is enlarged according to the display ratio corresponding to the relative size in the display screen, so that the area occupied by the graphic element in the shooting area is similar to the area occupied by the mobile phone screen W4 of the mobile phone 100, thereby improving user interactivity. At this time, the display angle is the angle between the central axis Z3 passing through the center point Q1 of the "apple" graphic element as shown in FIG4A and the central axis Z2 of the computer 200 screen. When the plane of the computer 200 screen and the plane of the mobile phone 100 screen are parallel, Z3 is parallel to the central axis Z1 of the mobile phone 100.

[0081] Specifically, the computer 200 can calculate the rotation angle and relative size of the captured shooting area relative to the actual display screen of the computer 200 in the X'OY' coordinate system based on the relevant data of the first screen background (i.e., the screen corresponding to the shooting area) in the XOY coordinate system of the mobile phone 100 and the relevant data of the actual display screen in the current X'OY' coordinate system, so as to obtain the display angle and display ratio that need to be adjusted when the graphic element is placed.

[0082] For example, in some embodiments, the mobile phone 100 can send the feature points of the first screen background in the XOY coordinate system to the computer 200, and the computer 200 calculates the display angle and display ratio corresponding to the placement of the graphic element based on the feature points of the first screen background. Among them, the feature points can reflect the significant information of the first screen background, and the values ​​of the feature points may include pixel values, coordinates, scales (Gaussian parameters), main directions, and directional gradient descriptors. For example, Figure 4B shows the first screen background B1 corresponding to the mobile phone screen W4 in Figure 4A. At this time, the feature points of the first screen background B1 can be extracted and sent to the computer 200. For example, there are 1,000 feature points of the first screen background, of which one feature point can be the pixel point information representing the "basket" located at the edge of the first screen background, and one feature point can be the pixel point information representing the center of the first screen background, etc.

[0083] It can be understood that the feature point information of the first background image is smaller than the original image data to be sent, so transmitting the feature point information can effectively save the amount of sent data, thereby reducing transmission power consumption.

[0084] It is understandable that in some embodiments, when the user uses the screen of the mobile phone 100 to shoot the screen of the computer 200, the user can adjust the relative size of the image element in the screen display and the screen of the computer 200 by zooming in / out of the mobile phone 100. Specifically, when the user moves the screen of the mobile phone 100 away from the screen of the computer 200, the screen of the computer 200 will become smaller in the image displayed on the screen of the mobile phone 100, and since the size of the image element remains unchanged, the image element will become larger relative to the screen of the computer 200; for another example, when the user moves the screen of the mobile phone 100 close to the screen of the computer 200, the screen of the computer 200 will become larger in the image displayed on the screen of the mobile phone 100, and since the size of the image element remains unchanged, the image element will become smaller relative to the screen of the computer 200.

[0085] For example, assuming that the shooting distance corresponding to mobile phone screen W5 in Figure 4C is h2, Figure 4C shows the mobile phone screens at different relative distances (specifically, distances h1 and h2 are shown in the figure), with the mobile phone 100 and computer 200 at the same relative angle in space. Mobile phone screens W4 and W5 correspond to distances h1 and h2 between the mobile phone 100 and computer 200 screens. Because distance h1 is greater than distance h2, the shooting area sq2 captured at distance h1 is larger than the shooting area sq3 captured at distance h2. Therefore, the size of the image primitive "apple" C1 relative to the computer 200 screen is larger at distance h1 than at distance h2. Therefore, as shown in Figure 4C, when the computer 200 screen displays the "apple" image primitive, the "apple" on the computer 200 screen corresponding to mobile phone screen W4 is larger than the "apple" on the computer 200 screen corresponding to mobile phone screen W5.

[0086] It can be understood that the display method mentioned in this application can be applied not only to the scenario mentioned above where posters need to be made, but also to any other scenario where other applications need to add image materials, and there is no limitation here.

[0087] It can be understood that in the display method mentioned in the present application, in addition to transferring graphic elements to the computer 200, the mobile phone 100 can also transfer the entire image to the computer 200. At this time, the mobile phone 100 does not need to perform graphic element operations, and directly displays the image captured or obtained from the gallery at a preset position on the screen of the mobile phone 100 according to a certain size. Then, the screen of the mobile phone 100 is photographed while displaying the image, so that the image on the computer 200 is displayed according to the corresponding display ratio and display angle.

[0088] It will be understood that the first electronic device and the second electronic device mentioned in this application may be any electronic device with a screen display function, including but not limited to mobile phones, tablet computers, vehicle-mounted equipment, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), servers, etc. The embodiments of the present invention do not impose any restrictions on the first electronic device and the second electronic device.

[0089] It can be understood that it has been mentioned above that the screen image captured by the mobile phone without superimposed graphics is called the "first screen background". At this time, for the convenience of explanation, the actual screen image of the computer 200 is called the "second screen background".

[0090] The display process will be described in detail below by taking the first electronic device as a mobile phone 100 and the second electronic device as a computer 200 as an example.

[0091] FIG5 shows a schematic diagram of a process in which, after a mobile phone 100 acquires an image element and uses a camera to shoot a picture of the screen of a computer 200, the image element can be displayed on the screen of the computer 200 according to a display angle and display ratio, according to some embodiments of the present application. The specific process includes:

[0092] S501, the mobile phone 100 generates metadata.

[0093] In some embodiments, the mobile phone 100 can obtain an image through a gallery or other channels such as photography, invoke a cutout algorithm to extract a salient area, thereby generating metadata, and display the generated metadata at a preset position on the computer 200, i.e., the center point of the metadata corresponds to the preset center position. Furthermore, the mobile phone 100 sends the metadata displayed at the preset position on the computer 200 to the computer 200.

[0094] In some implementations, when there are multiple salient areas on an image, the mobile phone 100 may extract multiple image primitives corresponding to the multiple salient areas in the image, or extract multiple image primitives from an area selected by the user in the image, generate multiple image primitives, and display the generated multiple image primitives at a preset location on the computer 200. Furthermore, the mobile phone 100 may send the multiple images displayed at the preset location on the computer 200 to the computer 200.

[0095] In other implementations, when there are multiple salient areas on an image, the mobile phone 100 can extract multiple image elements, generate multiple image element metadata, and then display them on the screen for the user to select the desired image element. The mobile phone 100 then redisplays the selected image element at a preset location on the computer 200. The mobile phone 100 then sends the image element displayed at the preset location on the computer 200 to the computer 200.

[0096] Furthermore, in some implementations, when there are multiple salient areas in an image, the mobile phone 100 may be set to extract the most salient area by default, generate one metadata, and then display the generated metadata at a preset location on the computer 200. Furthermore, the mobile phone 100 may send the metadata displayed at the preset location on the computer 200 to the computer 200.

[0097] For example, as shown in FIG6A , a schematic diagram illustrates a mobile phone 100 capturing a foreground image and displaying a primitive. As shown in FIG6A , on mobile phone screen W11, the user selects the "shoot button" P1, activating the camera function of mobile phone 100. The user then uses mobile phone 100 to photograph an apple and a bowl on a table. When the user taps the screen of mobile phone 100 to confirm the capture, mobile phone screen W12 shown in FIG6A is the captured image after the user taps the screen. The image corresponding to this captured image is the foreground image. At this point, mobile phone 100 can use a cutout algorithm to extract primitives. Assuming the default settings extract the primitives for the most prominent region, the apple region in mobile phone screen W12 can be extracted to generate an "apple" primitive C1. This generated "apple" primitive C1 is then displayed at the center of the mobile phone 100 screen, as shown in FIG6A , on mobile phone screen W13. At this point, the center point of "apple" primitive C1 coincides with the center position. 6C , the mobile phone 100 sends the “apple” graphic element C1 corresponding to the mobile phone screen W13 shown in FIG6A as the first screen foreground to the computer 200 , so that the computer 200 can subsequently adjust and display the “apple” graphic element C1 .

[0098] For example, FIG6B shows a schematic diagram of the mobile phone 100 acquiring a foreground image through the gallery. As shown in the mobile phone screen W21 in FIG6B , the user clicks the “Gallery” button P2, and the mobile phone 100 can enter the gallery. As shown in FIG6B , six images are displayed on the mobile phone screen 22, and the user clicks on an image in the upper left corner. At this time, the image clicked by the user is displayed on the mobile phone as the foreground image. For example, in the mobile phone screen 23 in FIG6B , the mobile phone 100 displays the image clicked by the user in the upper left corner of the mobile phone screen W22 as the foreground image. It can be understood that after the mobile phone 100 acquires the foreground image, the process of extracting and displaying the graphic elements is essentially the same as the process of extracting and displaying the graphic elements after the foreground image is captured as described above, and will not be elaborated here.

[0099] S502 , the mobile phone 100 takes a photo of the screen of the computer 200 at a relative position desired by the user, wherein the relative position includes a relative angle and a relative distance.

[0100] It is understandable that the mobile phone 100 can be directed to the screen of the computer 200 while displaying the screen of the graphic element, and then adjust the relative angle and relative distance between the mobile phone 100 and the computer 200 during the shooting process (for example, rotation, zooming in / out, etc.), thereby adjusting the specific center position, relative size, and rotation angle of the graphic element on the screen of the mobile phone 100 relative to the captured screen of the computer 200, thereby achieving the desired graphic element placement effect. After detecting that the current shooting effect is the user's desired effect, the mobile phone 100 sends the first screen background related data (for example, the feature points of the first screen background obtained by the feature extraction algorithm) to the computer 200.

[0101] Specifically, in some embodiments, when the camera function of the mobile phone 100 is used to shoot the computer 200, the mobile phone 100 will obtain the first screen background. For example, Figure 6D shows the corresponding situation of the screen of the computer 200 displayed on the screen of the mobile phone 100. When the user uses the mobile phone 100 to shoot the screen of the computer 200 (i.e., the second screen background), the three-dimensional space content will be mapped to the two-dimensional plane, and the picture displayed on the screen of the mobile phone 100 will correspond to the partial screen content of the screen of the computer 200. Specifically, there is a spatial mapping relationship between the partial content on the first screen background and the partial content on the second screen background. As shown in the figure, the pixel point rt1, pixel point rt2, and pixel point rt3 on the screen of the computer 200 (second screen background) correspond to the pixel point r1, pixel point r2, and pixel point r3 on the screen of the mobile phone 100 (first screen background), respectively.

[0102] In order to obtain the desired graphic element placement effect, when the mobile phone 100 is shooting, the graphic element will be displayed on the screen of the mobile phone 100, and the user uses the mobile phone 100 displaying the graphic element to shoot the computer 200. At this time, the graphic element will be superimposed on the photographed computer 200.

[0103] It is understandable that when the mobile phone 100 is photographing the computer 200, the user can adjust the relative distance and relative angle of the mobile phone 100 (such as rotating, zooming in / out, tilting, etc.) so that the image element on the screen of the mobile phone 100 is placed at the desired position on the screen of the computer 200 being photographed. When the user performs the preset "end photographing" operation, the mobile phone 100 detects the user's "end photographing" operation instruction, extracts feature points from the image data of the first screen background in the displayed image using a feature extraction algorithm, and sends the extracted first feature points to the computer 200, wherein the first feature points can reflect the content information of the first screen background.

[0104] It is understandable that the first feature point information is smaller than the original image data, effectively saving the amount of data sent. Among them, the feature extraction algorithm can be a variety of scale-invariant feature transform (SIFT), local binary patterns (LBP) algorithm, feature extraction algorithm (histogram of oriented gradient, HOG), etc. It is understandable that the value of the feature point can include pixel value, coordinates, scale (Gaussian parameter), main direction, and directional gradient descriptor.

[0105] For example, from the mobile phone screen W3 to the mobile phone screen W4 in FIG4A , the user rotates the mobile phone 100, and corresponding to the mobile phone screen W4, the user clicks the screen to perform the preset "end shooting" operation. After the mobile phone 100 detects the user's "end shooting" operation instruction, the mobile phone 100 performs shift feature point extraction on the first screen background B1 as shown in FIG4B to obtain the first feature point, and sends the first feature point to the computer 200. The first feature point includes significant information of the first screen background B1, such as the size of the first screen background B1, and the captured information corresponding to the content of the computer 200 screen (such as a basket, etc.). It can be understood that the feature point data of the first screen background B1 of the mobile phone 100 is represented according to the data representation method of the coordinate system of the mobile phone 100.

[0106] It is understandable that the above predetermined operation realizes the shooting process. In addition to the user clicking the screen, the shooting process can also be realized by using physical buttons (such as the back button, volume button, power button), etc. The long press can also be changed to single click, double click, press or slide, etc. The specific predetermined operations are not detailed here.

[0107] It is understandable that in other embodiments, when the mobile phone 100 is shooting at the computer 200, the user can adjust the relative distance and relative angle between the mobile phone 100 and the computer 200. The user does not need to perform the predetermined operation at this time. After the mobile phone 100 automatically detects that it is aimed at the computer 200 (for example, the user remains stationary at the current position for more than a preset number of seconds), it will realize shooting of the computer 200.

[0108] S503: The computer 200 performs screen background image matching.

[0109] It is understandable that there will be a large gap between the pixel data features of the first screen background and the screen content of the computer 200 obtained by shooting the computer 200 with the mobile phone 100 and the actual display screen of the second screen background on the computer 200. At this time, it is necessary to perform feature matching on the first screen background and the second screen background shot by the mobile phone 100, and find pixel points from the second screen background that are relatively similar to the first feature points of the first screen background, so as to obtain feature points (for the convenience of explanation, referred to as "second feature points") used to represent the pixel points that are relatively similar to the features found. A set of registration points is obtained based on the first feature points and the second feature points, and there is a one-to-one correspondence between the first feature points and the second feature points in the registration point set.

[0110] In some embodiments, a matching algorithm, such as a k-nearest neighbor (kNN) algorithm, can be used to screen the first feature points of the first screen background and each pixel point in the second screen background to find pixels with similar features, thereby obtaining second feature points. The first feature points of the first screen background and the second feature points found in the second screen background are used as a set of registration points. For example, the first feature points of the first screen background are: ST{ST1, ST2, ST3, ..., STN}; the second feature points of the second screen background are: DT{DT1, DT2, DT3, ..., DTN}.

[0111] For example, Figure 6E shows pixels S1, S2, and S3 corresponding to three feature points ST1, ST2, and ST3 in the first screen background of mobile phone 100, displayed on screen B2. Feature point ST1 corresponds to pixel S1 on the upper left edge of a basket, feature point ST2 corresponds to pixel S2 on the lower edge of the basket, and feature point ST3 corresponds to pixel S3 on the upper right edge of the basket. Furthermore, Figure 6E shows the pixels corresponding to feature points DT1, DT2, and DT3 on the second screen background 002D of computer 200, which match feature points ST1, ST2, and ST3. Feature point DT1 corresponds to pixel point D1 on the upper left edge of the basket, feature point DT2 corresponds to pixel point D2 on the lower edge of the basket, and feature point DT3 corresponds to pixel point D3 on the upper right edge of the basket. It can be understood that since the features of ST1, ST2, and ST3 correspond to the features of DT1, DT2, and DT3, the three pixel points S1, S2, and S3 in the background of the first screen correspond one-to-one to the three pixel points D1, D2, and D3 in the background of the second screen.

[0112] S504, the computer 200 determines whether the matching is successful.

[0113] It is understandable that when the mobile phone 100 is photographing the computer 200 , the background of the first screen captured may not necessarily contain feature points that match the pixel data of the second screen background.

[0114] In some embodiments, when the computer 200 matches the first feature point of the first screen background with the pixel data of the second screen background, if no pixel with similar features is found in the pixel data of the second screen background, the computer 200 determines that the match is unsuccessful and proceeds to S502, requiring the mobile phone 100 to re-interact between screens and reacquire the relevant data of the first screen background that can be used for matching. If the computer 200 finds a pixel with similar features in the pixel data of the second screen background and determines that the match is successful, the computer 200 proceeds to S505, where the computer 200 adjusts the primitives for display.

[0115] S505: The computer 200 displays the graphic element.

[0116] In some embodiments, a spatial mapping relationship between the first screen background and the second screen background is obtained based on the set of registration points. It is understandable that the spatial mapping relationship corresponds to the relative position between the mobile phone 100 and the computer 200. It is understandable that based on the obtained spatial mapping relationship, the corresponding position of any pixel on the screen of the computer 200 captured by the mobile phone 100 on the actual screen of the computer 200 can be obtained. At this time, based on the spatial mapping relationship, the corresponding center position of the center point of the primitive in the second screen background, the display ratio and display angle of the primitive when displayed on the screen of the computer 200 can be obtained. Then, the primitive is displayed at the corresponding center position according to the display ratio and display angle. The specific process of obtaining the display ratio and display angle required for the primitive based on the spatial mapping relationship is described in Figure 7 below and will not be repeated here.

[0117] For the sake of convenience, the corresponding point of the center point of the graphic element in the first screen background is called the "first center point", and the corresponding point of the center point of the graphic element in the second screen background is called the "second center point".

[0118] Specifically, after the computer 200 obtains the first center point of the first screen background captured by the mobile phone 100, the spatial mapping relationship is used to find the second center position corresponding to the first center position of the first center point on the first screen background on the actual screen of the computer 200 (i.e., the second screen background). After the received graphic element is displayed on the screen of the computer 200 according to the display angle and display ratio required by the spatial mapping relationship, the graphic element is placed at the second center position using Alpha blending and overlay, and edge filtering optimization is performed at the same time to obtain the final graphic element display effect.

[0119] For example, assuming that the first center position of the first center point of the "apple" graphic element on the mobile phone 100 is (a1, b1) in the XOY coordinate system, at this time, according to the spatial mapping relationship, the second center position of the first center point in the X'OY' coordinate system of the computer 200 screen is (a1', b1'), the display angle of the graphic element is θ1, and the display ratio is k1. The "apple" graphic element C1 received from the mobile phone 100 is displayed at a display angle of θ1 and scaled according to a display ratio of k1, and is placed at the second center position (a1', b1') through Alpha blending and overlay technology.

[0120] It can be understood that, for example, as shown in 6F, after the Alpha blending and overlaying technology, the "apple" graphic element C1 in (1) of Figure 6F will have some burrs and distortions. By using the edge filtering technology to optimize, as shown in (2) of Figure 6F, a smooth "apple" graphic element C1 is obtained.

[0121] It can be understood that the execution order of the above steps S501 to S505 is only an illustration. In other embodiments, other execution orders may be adopted, and some steps may be split or combined, which is not limited here.

[0122] As can be understood, the present application obtains image-generating primitives through a first electronic device (e.g., mobile phone 100), and through convenient screen-to-screen interaction, the primitives can be accurately placed on the screen of a second electronic device (e.g., computer 200) according to a specific center position, display ratio, and display angle, simplifying the interaction process. Since the first electronic device only needs to transmit the primitive data and the first screen background-related data (e.g., the first feature point of the first screen background) to the second electronic device, the interaction process between devices is simplified and the amount of data transmitted between devices is optimized.

[0123] FIG7A illustrates a schematic diagram of a specific process for determining the display ratio and display angle that need to be adjusted when placing a graphic element on the screen of computer 200 based on the spatial mapping relationship between the first screen background and the second screen background, according to some embodiments of the present application. Specifically, as shown in FIG7A , the specific process may include:

[0124] S701: Obtain a spatial mapping relationship between a first screen background and a second screen background according to a set of registration points.

[0125] In some embodiments, from the registration point sets of the first screen background and the second screen background, pairs of registration points with high similarity are selected to construct a spatial transformation matrix (e.g., a homography matrix) for perspective transformation, and the spatial transformation matrix is ​​used as a spatial mapping relationship. It will be understood that the two planes of the first screen background and the second screen background are registered in space to find a spatial mapping relationship, which is related to spatial properties such as the relative distance and relative angle between the mobile phone 100 and the computer 200 in space.

[0126] S702, selecting a first center point St1 and a first offset point St2 from the first background screen, and calculating the corresponding second center point Dt1 and second offset point Dt2 in the second background screen through the spatial mapping relationship between the first screen background and the second screen background.

[0127] It can be understood that the first offset point is any pixel point in the first background screen corresponding to the screen of the computer 200 photographed. For example, it can be any pixel point on the screen of the computer 200 photographed on the first background screen that is superimposed by the apple.

[0128] In some embodiments, a first center point St1 and a second center point Dt1 and a second offset point Dt2 corresponding to a first offset point St2 in the second background screen are obtained according to a spatial transformation matrix (eg, a homography matrix).

[0129] For example, FIG. 7B shows a first center point St1 and a first offset point St2 corresponding to a second center point Dt1 and a second offset point Dt2 in the second background screen.

[0130] S703 , calculating the display ratio and display angle corresponding to the graphic element according to the first center point St1 , the first offset point St2 , the second center point Dt1 , and the second offset point Dt2 .

[0131] In some embodiments, for the required display ratio of a graphic element, the following steps are performed: The coordinates (st1_x, st1_y) and (st2_x, st2_y) of the first center point St1 and the first offset point St2 are obtained, and the Euclidean distance d1 between the first center point St1 and the first offset point St2 is calculated based on the coordinates; the coordinates (dt1_x, dt1_y) and (dt2_x, dt2_y) of the second center point Dt1 and the second offset point Dt2 are obtained, and the Euclidean distance d2 between the second center point Dt1 and the second offset point Dt2 is calculated. It will be understood that the ratio of d2 to d1 is the ratio of the size of the graphic element displayed on the screen by computer 200 to the size of the graphic element received by computer 200. This ratio of d2 to d1 is used as the display ratio for the shared graphic element.

[0132] In some embodiments, for the display angle required by the graphic element: obtain the coordinates (st1_x, st1_y) and (st2_x, st2_y) of the first center point St1 and the first offset point St2, and calculate the angle β1 between the straight line where the first center point St1 and the first offset point St2 are located and OY based on the coordinates; obtain the coordinates (dt1_x, dt1_y) and (dt2_x, dt2_y) of the second center point Dt1 and the second offset point Dt2, and calculate the angle β2 between the straight line where the second center point Dt1 and the second offset point Dt2 are located and OY' based on the coordinates, and the difference between the two angles |β1-β2| is used as the display angle shared by the graphic element.

[0133] For example, in Figure 7B , after computer 200 receives the metadata, the OY axis in mobile phone 100's coordinate system XOY is parallel to OY' in computer 200's coordinate system X'OY'. At this point, the angle between the line containing the first center point St1 and the first offset point St2 and OY is calculated for the mobile phone 100 interface to obtain Angle 1. The angle between the line containing the second center point Dt1 and the second offset point Dt2 and OY' is calculated for the computer 200 interface to obtain Angle 2. Dividing Angle 1 by Angle 2 yields the display angle.

[0134] It can be understood that the execution order of the above steps S701 to step 703 is only an illustration. In other embodiments, other execution orders may be adopted, and some steps may be split or combined, which is not limited here.

[0135] Figure 8 illustrates a schematic diagram of the structural modules of a mobile phone 100 and computer 200, according to some embodiments of the present application, that enable computer 200 to display adjusted graphics primitives on its screen. Specifically, as shown in Figure 8 , mobile phone 100 includes an application 801A and a graphics primitive calculation unit 802A. Computer 200 includes an application 801B and a graphics primitive calculation unit 802B. Graphics primitive calculation units 802A and 802B cooperate to implement the display method described above in Figure 5 .

[0136] Specifically, application 801A in mobile phone 100 is an application that displays graphics elements and has camera functions, for example, the material sharing software shown in Figure 3A. Application 801B in computer 200 is an application that can display received graphics elements, for example, the image processing software shown in Figure 3A. It can be understood that application 801A in mobile phone 100 and application 801B in computer 200 can each call the graphics element calculation unit in each device, so that computer 200 can use mobile phone 100 to obtain the graphics elements to be displayed and display the graphics elements in the corresponding positions.

[0137] The primitive calculation unit 802A includes a connection subunit 8021A, a primitive generation subunit 8022A, and an inter-screen interaction subunit 8023A.

[0138] The primitive calculation unit 802B includes a connection subunit 8021B, a matching subunit 8022B, a primitive display subunit 8023B, and a data storage subunit 8024B.

[0139] The connection subunit 8021A and the connection subunit 8021B perform service initialization and device connection. The primitive generation subunit 8022A is used to generate primitive metadata. For example, it can execute the detailed description of S501 in Figure 5 above, which will not be repeated here. At this time, the primitive generation subunit 8022A will send the primitive metadata to the computer 200, and the data storage subunit 8024B in the computer 200 will store the received primitive metadata.

[0140] The inter-screen interaction sub-unit 8023A is configured to display graphics on the screen and use the camera function to capture the first screen background of the computer 200. The graphics on the screen of the mobile phone 100 are superimposed on the first screen background. Furthermore, the acquired data related to the first screen background is sent to the computer 200. The data storage sub-unit 8024B in the computer 200 stores the received data related to the first screen background. For example, the detailed description of S502 in FIG. 5 can be performed, and will not be repeated here.

[0141] The matching subunit 8022B is used to match the stored first screen background related data with the second screen background related data to determine whether the match is successful. For example, the contents described in detail in S503 and S504 in Figure 5 above can be executed and will not be repeated here.

[0142] The graphic element display subunit 8023B is configured to obtain a spatial mapping relationship between the first screen background and the second screen background, and based on the spatial mapping relationship, obtain a display angle, a display scale, and a corresponding second center position corresponding to the graphic element, and adjust the graphic element for display. For example, the specific description of S505 in FIG. 5 may be performed, and will not be repeated here.

[0143] It can be understood that the structural modules of the mobile phone 100 and the computer 200 shown in Figure 8 are only an example. In other embodiments, other module structures can also be used, and the embodiments of the present application are not limited thereto.

[0144] It is understood that the mobile phone 100 and the computer 200 can also be connected, or the computer 200 can provide image materials and display the required graphics on the screen of the mobile phone 100. In this case, the mobile phone 100 also includes a matching sub-unit 8022B and a graphics display sub-unit 8023B; correspondingly, the computer 200 also includes a graphics generation sub-unit 8022A and an inter-screen interaction sub-unit 8023A, which are not required here.

[0145] It is understandable that the applications mentioned in this application that support primitive display in the computer 200 and the applications that support primitive acquisition and primitive transmission on the mobile phone 100 are not only the image processing software and material sharing software mentioned above, but can also be any other applications that support this function, such as office applications, social software applications, specifically memos, notes, instant messaging software, etc., all of which are within the scope of protection of this application. In addition, it is understandable that the primitive calculation unit in the mobile phone 100 and the computer 200 in Figure 8 can be included in the application, or it can be a functional unit provided by the system in the mobile phone 100 and the computer 200. When the application of the mobile phone 100 and the computer 200 needs to use the primitive calculation functional unit, the application in the mobile phone 100 and the computer 200 can call the corresponding functional module from the system to cooperate in realizing primitive display.

[0146] It is understandable that when the user shares image elements on the same display screen of the image processing software on the computer 200 multiple times, the background of the second screen on the computer 200 will not change.

[0147] It can be understood that there is no strict master-slave role relationship between the first electronic device (such as mobile phone 100) and the second electronic device (such as computer 200), which facilitates the flexible implementation of image element sharing services between devices (i.e., the image elements sent by the first electronic device are displayed on the screen of the second electronic device).

[0148] Furthermore, FIG9 shows a schematic diagram of the software structure of a mobile phone 100 according to some embodiments of the present application. As shown in FIG9 , the software structure of the mobile phone 100 is a multi-layer structure, which includes, from top to bottom, an application layer 91, an application framework layer 92, a system library 93, and a kernel layer 94.

[0149] The application layer 91 may include a series of application packages. As shown in FIG9 , the application layer 91 includes applications such as Bluetooth 910, Music 911, Gallery 912, Camera 913, and Material Sharing 914. In some embodiments of the present application, Material Sharing 914 may provide image primitives for image materials that the user wishes to display on the computer 200.

[0150] The application framework layer 92 provides an application programming interface (API) and programming framework for applications in the application layer 91. The application framework layer includes some predefined functions. The application framework layer 92 includes a notification manager 920, a view system 921, a content provider 922, a primitive calculation 923, a resource manager 924, and the like. In some embodiments of the present application, the primitive calculation 923 is used to provide the material sharing 914 with an API and programming framework for calling primitive-related operations in the system library.

[0151] The system library 93 is the core of the mobile phone 100 software system and can provide services to applications in the application layer 91 through the application framework layer 92. The system service layer 93 includes a media library 930, a graphics processor 931, a surface management library 932, a signal processing library 933, and a primitive calculation library 934. In some embodiments of the present application, the primitive calculation library 934 can receive instructions from the primitive calculation 923 and perform primitive processing-related operations. For example, the primitive calculation library 934 includes the primitive calculation unit 802A in Figure 8, which performs operations such as primitive generation and screen interaction.

[0152] The kernel layer 94 includes a display driver 940, a camera driver 941, a network card driver 942, an audio driver 943, and a sensor driver 944. In some embodiments of the present application, the camera driver 941 can correspond to the camera function and drive the camera to shoot.

[0153] It is understood that FIG9 shows only an example of the software structure of the mobile phone 100. In other embodiments, the mobile phone 100 may also adopt other forms of software structures, such as Android TM 、iOS TM 、HarmonyOS TM 、Tizen TM The present invention is not limited to the embodiments of the present invention.

[0154] FIG10 shows a schematic diagram of the hardware structure of a mobile phone 100 according to an embodiment of the present application.

[0155] In Figure 10 , similar components have the same reference numerals. As shown in Figure 10 , the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0156] It should be understood that the illustrated structure of the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown, or some components may be combined or separated, or arranged differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0157] The processor 110 may include one or more processing units, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a microprocessor (MCU), an artificial intelligence (AI) processor, or a processing module or processing circuit such as a programmable logic device (FPGA). Different processing units may be independent devices or integrated into one or more processors. In some embodiments, the processor 100 may perform the operations in the display method related to the mobile phone 100 in Figure 5 above.

[0158] The memory can be used to store data, software programs and modules, and can be a volatile memory (volatile memory), such as random-access memory (RAM); or a non-volatile memory (non-volatile memory), such as read-only memory (ROM), flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, or it can be a removable storage medium, such as a secure digital (SD) memory card.

[0159] The charging management module 140 may include a power supply, a power management component, and the like. The power supply may be a battery. The power management component manages the charging of the power supply and its supply to other modules. The charging management module receives charging input from a charger; the power management module connects the power supply to the charging management module and the processor 110.

[0160] The mobile communication module 150 may include, but is not limited to, an antenna, a power amplifier, a filter, a low noise amplifier (LNA), and the like. The mobile communication module 150 may provide wireless communication solutions, including 2G / 3G / 4G / 5G, for the mobile phone 100. The wireless communication module 160 may include an antenna, and transmits and receives electromagnetic waves via the antenna.

[0161] The mobile phone 100 can communicate with the network and other devices through wireless communication technology.

[0162] In some embodiments, the mobile communication module 150 and the wireless communication module 160 of the mobile phone 100 may also be located in the same module.

[0163] Camera 193 is used to capture still images or video. The optical image of a scene is projected onto a photosensitive element through the lens. The photosensitive element converts the optical signal into an electrical signal, which is then passed to an image signal processor (ISP) for conversion into a digital image signal. Mobile phone 100 utilizes the ISP, camera 193, video codec, graphics processing unit (GPU), display, and application processor to implement its camera function.

[0164] The display screen includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a Micro LED, a Micro OLED, or a quantum dot light-emitting diode (QLED).

[0165] The audio module 170 can convert digital audio information into an analog audio signal output, or convert analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0166] The interface module includes an external memory interface, a universal serial bus (USB) interface, and a subscriber identification module (SIM) card interface. The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 via the external memory interface to implement data storage. The USB interface is used for the mobile phone 100 to communicate with other mobile phones. The subscriber identification module card interface is used to communicate with the SIM card installed in the mobile phone 100, for example, to read the phone number stored in the SIM card or write the phone number to the SIM card.

[0167] In some embodiments, the mobile phone 100 further includes buttons, a motor, and an indicator. The buttons may include a volume button, an on / off button, and the like. The motor is used to vibrate the mobile phone 100. The indicator may include a laser indicator, a radio frequency indicator, an LED indicator, and the like.

[0168] It is understandable that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100 .

[0169] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by the hardware integrated logic circuit in the processor or by instructions or program codes in the form of software. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0170] According to the method provided in the embodiments of the present application, the present application also provides a computer program product, which includes: computer program code, which, when executed on a computer, enables the computer to implement the steps performed by the electronic device in any one of the above embodiments.

[0171] According to the method provided in the embodiments of the present application, the present application also provides a computer-readable medium, which stores program code. When the program code runs on a computer, the computer implements the steps performed by the electronic device in any of the above embodiments.

[0172] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0173] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0174] Program code can be implemented with a high-level programming language or an object-oriented programming language to communicate with the processing system. Where necessary, program code can also be implemented in assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0175] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including but not limited to a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic, or other form of propagation signal. Therefore, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0176] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and in some embodiments, such features may not be included or may be combined with other features.

[0177] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems raised by this application. In addition, in order to highlight the innovative part of this application, the above-mentioned device embodiments of this application do not introduce units / modules that are not closely related to solving the technical problems raised by this application. This does not mean that other units / modules do not exist in the above-mentioned device embodiments.

[0178] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "including a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0179] Although the present application has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the application.

Claims

1. A display method, characterized in that: include: The first electronic device detects a first sharing operation of a user to share first information with a second electronic device, and the first electronic device captures a display screen of the second electronic device with first location information; The second electronic device displays the first information using a first display parameter; The first electronic device detects a second sharing operation of a user to share second information with a second electronic device, and the first electronic device captures a display screen of the second electronic device with second location information; The second electronic device displays the second information using a second display parameter; The first position information and the second position information include at least one of a relative distance and a relative angle between the first electronic device and the second electronic device, and the first display parameter and the second display parameter include at least one of a display ratio and a display angle.

2. The method according to claim 1, characterized in that The first information and the second information are images or picture elements in the images.

3. The method according to claim 2, characterized in that The display ratio is a ratio between a size of the first information displayed on the second electronic device and a size of the first information displayed on the first electronic device.

4. The method according to claim 3, characterized in that The first position information includes a first relative distance, and the first display parameter includes a first display ratio; The second position information includes a second relative distance, and the second display parameter includes a second display ratio, wherein The first relative distance is smaller than the second relative distance, and the first display ratio is smaller than the second display ratio.

5. The method according to claim 1, characterized in that Also includes: The first position information includes a first relative angle, and the first display parameter includes a first display angle; The second position information includes a second relative angle, the second display parameter includes a second display angle, Among them, the first relative angle is smaller than the second relative angle, and the first display angle is smaller than the second display angle.

6. The method according to claim 1, characterized in that The first information includes a first picture element in the first image, and the method further includes: After detecting the first sharing operation, the first electronic device extracts the first image or a plurality of graphic elements in a first area selected by a user in the first image; The first electronic device displays the plurality of graphic elements, and transmits a first graphic element selected by a user from the plurality of graphic elements to the second electronic device.

7. The method according to claim 1, characterized in that The first information includes a first picture element in the first image, and the method further includes: After detecting the first sharing operation, the first electronic device extracts a graphic element from the first image; The first electronic device displays the extracted graphic element as a first graphic element, and sends it to the second electronic device.

8. The method according to claim 1, characterized in that The first information is displayed in a first application interface of the second electronic device.

9. The method according to claim 1, characterized in that: The method further comprises: The first electronic device sends, to the second electronic device, first screen background image information of a first screen background image obtained by photographing the display screen of the second electronic device using the first location information; The second electronic device determines the first display parameter based on the first screen background image information.

10. The method according to claim 9, characterized in that The first screen background image information includes a plurality of first image feature points of the first screen background image; and The second electronic device determines the first display parameter based on the first screen background image information, including: The second electronic device determines the first display parameter based on the multiple first image feature points of the first screen background image and multiple second image feature points corresponding to the multiple first image feature points in the first screen image of the current display interface of the second electronic device.

11. The method according to claim 10, characterized in that The first display parameter includes the first display ratio, and The second electronic device determines the first display parameter based on the multiple first image feature points of the first screen background image and multiple second image feature points corresponding to the multiple first image feature points in the first screen image of the current display interface of the second electronic device, including: Obtaining a spatial transformation matrix according to the plurality of first image feature points and the plurality of second image feature points; Determine, from the first screen image of the current display interface of the second electronic device according to the spatial transformation matrix, a second center point and a second offset point corresponding to the first center point and the first offset point of the first screen background image; A first distance is calculated according to the coordinates of the first center point and the coordinates of the first offset point, and a second distance is calculated according to the coordinates of the second center point and the coordinates of the second offset point; The first display ratio is obtained according to a ratio between the first distance and the second distance.

12. The method according to claim 10, characterized in that The first display parameter includes the first display angle, and The second electronic device determines the first display parameter based on the multiple first image feature points of the first screen background image and multiple second image feature points corresponding to the multiple first image feature points in the first screen image of the current display interface of the second electronic device, including: Obtaining a spatial transformation matrix according to the plurality of first image feature points and the plurality of second image feature points; Determine, from the first screen image of the current display interface of the second electronic device according to the spatial transformation matrix, a second center point and a second offset point corresponding to the first center point and the first offset point of the first screen background image; Calculating a first angle between a straight line where the first center point and the first offset point are located and a first coordinate axis in a coordinate system of a second electronic device, and calculating a second angle between a straight line where the second center point and the second offset point are located and the first coordinate axis in the coordinate system of the second electronic device; The first display angle is obtained according to the difference between the first angle and the second angle.

13. An electronic device, characterized in that: include: one or more processors; One or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device executes the display method described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The storage medium stores instructions, which, when executed on a computer, cause the computer to execute the display method according to any one of claims 1 to 12.

15. A computer program product comprising instructions, characterized in that When the computer program product is executed on a computer, the computer is enabled to execute the display method according to any one of claims 1 to 12.

Images