Display method, medium and electronic equipment

Abstract

The invention relates to the technical field of computers, in particular to a display method, a medium and electronic equipment. In the display method, when a specific relative angle and a specific relative distance exist between first electronic equipment and second electronic equipment, a shot picture of a user wanted primitive at a specific placement position of a screen of the second electronic equipment is displayed on the screen of the first electronic equipment; and at the moment, displaying the primitives received from the first electronic equipment on a screen of the second electronic equipment, and displaying the primitives according to the corresponding display angle and display size. It can be understood that when the second electronic device displays according to the corresponding display angle and display size, the placement condition of the primitives displayed on the second electronic device meets the effect that the rotation angle and the relative size of the primitives on the shot picture on the first electronic device relative to the screen area of the second electronic device are achieved; and therefore, the interactivity of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a display method, a medium and an electronic device. Background Art

[0002] Currently, data can be shared between various electronic devices. For example, data such as images and text can be shared between devices through common Bluetooth, wireless networks, etc. When sharing images between electronic devices, users can use electronic devices to share complete images or only share image primitives (e.g., specific elements such as people and objects in the image).

[0003] For example, Figure 1 FIG. 1 shows a schematic diagram of a mobile phone 100 using a cloud 300 to share data with a computer 200, wherein the mobile phone 100 can transmit data with the computer 200 via the computer cloud 300. Figure 1 As shown in , when a user uses the image processing software (Photoshop, PS) on a computer to make a poster, the user wants to obtain some image materials. At this time, it is not convenient for the user to use the computer 200 to shoot the materials, and the user needs to use the mobile phone 100 to obtain the required image materials and transfer them to the computer 200. Sometimes, there is a lot of information on the image material, and the user wants to directly obtain certain specific image elements on the image material without some messy background information. The mobile phone 100 can extract the image elements from the image material and transmit them to the computer 200, so that the computer 200 can process the image elements.

[0004] Specifically, the user can use the mobile phone 100 to obtain Figure 1 , the mobile phone 100 can send the image 1 to the cloud 300. The intelligent cutout technology in the cloud 300 can extract the various primitives in the image 1, for example, extract the "apple" primitive C1 in the image 1, and then send it to the mobile phone 100; the mobile phone 100 will display the extracted primitives, for example, the "apple" primitive. Because multiple primitives are sometimes extracted, the user can select the primitive to be transmitted from the multiple primitives, for example, the user selects the "apple" primitive. At this time, the mobile phone 100 will send the "apple" primitive to the cloud 300, and the cloud 300 will forward it to the computer 200, so that the computer 200 will display it at a fixed position on the image processing software interface.

[0005] However, since the computer 200 can only display the received graphics in a fixed form and at a fixed position, the user cannot directly obtain the desired placement effect, and needs to adjust the placement of the graphics on the screen through other operations such as dragging, resulting in poor interactivity. Summary of the invention

[0006] In order to solve the above problems, the present application provides a display method, a medium, and an electronic device. In the display method, when there is a specific relative angle and a specific relative distance between a first electronic device and a second electronic device, after the screen of the first electronic device displays a shot of the specific placement position of the graphic element that the user wants on the screen of the second electronic device, the screen of the second electronic device displays the graphic element received from the first electronic device, and displays the graphic element at the corresponding display angle and display size. It can be understood that when the second electronic device displays at the corresponding display angle and display size, the placement of the graphic element displayed on the second electronic device achieves an effect consistent with the rotation angle and relative size of the graphic element on the shot screen on the first electronic device relative to the screen area of ​​the second electronic device, thereby improving user interactivity.

[0007] In a first aspect, an embodiment of the present application provides a display method, including: a first electronic device detects a first sharing operation of a user to share first information with a second electronic device, and the first electronic device captures the display screen of the second electronic device with first position information; the second electronic device displays the first information with first display parameters; the first electronic device detects a second sharing operation of a user to share second information with the second electronic device, and the first electronic device captures the display screen of the second electronic device with second position information; the second electronic device displays the second information with second display parameters; wherein the first position information and the second position information include at least one of a relative distance and a relative angle between the first electronic device and the second electronic device, and the first display parameter and the second display parameter include at least one of a display ratio and a display angle.

[0008] It is understandable that the user's sharing operation can be in various forms, for example, the user can perform a physical operation such as clicking on the screen of the first electronic device, or the user can perform an operation such as remaining still facing the screen of the second electronic device for a preset time.

[0009] It is understandable that the first electronic device detects the user's first sharing operation, and the first electronic device captures the display screen of the second electronic device corresponding to the first position information. The captured image includes the specific placement position of the first information relative to the captured second electronic device, such as the rotation angle and relative size of the first information relative to the screen of the second electronic device. It is understandable that the display ratio and display angle in the first display parameter correspond to the captured image in the first electronic device. At this time, when the second electronic device displays the first information with the corresponding first display parameter, since the first display parameter includes at least one of the display ratio and display angle, when the second electronic device displays the first information, the first information on the screen of the second electronic device will produce the effect of the rotation angle and relative size relative to the second electronic device when displayed on the first electronic device, thereby improving user interactivity.

[0010] In a possible implementation of the first aspect above, the first information and the second information are images or graphic elements in the images.

[0011] In a possible implementation of the first aspect, the display ratio is a ratio between a size of the first information displayed on the second electronic device and a size of the first information displayed on the first electronic device.

[0012] In a possible implementation of the first aspect above, the first position information includes a first relative distance, and the first display parameter includes a first display ratio; the second position information includes a second relative distance, and the second display parameter includes a second display ratio, wherein the first relative distance is smaller than the second relative distance, and the first display ratio is smaller than the second display ratio.

[0013] It is understandable that when the user uses the screen of the first electronic device to shoot the screen of the second electronic device, the relative size of the first information and the screen of the second electronic device in the screen display can be adjusted by zooming in / out on the first electronic device and changing the relative distance accordingly. When the screen of the first electronic device is far away from the screen of the second electronic device, the screen of the second electronic device in the picture displayed on the screen of the first electronic device will become smaller, and since the size of the first information remains unchanged, at this time, the first information will become larger relative to the screen of the second electronic device, so the display ratio is relatively large; for another example, when the user brings the screen of the first electronic device close to the screen of the second electronic device, the screen of the second electronic device in the picture displayed on the screen of the first electronic device will become larger, and since the size of the first information remains unchanged, at this time, the first information will become smaller relative to the screen of the second electronic device, so the display ratio is relatively small.

[0014] It is understandable that the user can adjust the size of the graphic element / image displayed on the second electronic device accordingly by adjusting the relative distance between the first electronic device and the second electronic device, thereby improving the user experience.

[0015] In a possible implementation of the first aspect above, it also includes: the first position information includes a first relative angle, and the first display parameter includes the first display angle; the second position information includes a second relative angle, and the second display parameter includes a second display angle, wherein the first relative angle is smaller than the second relative angle, and the first display angle is smaller than the second display angle.

[0016] It is understandable that when a user uses the screen of a first electronic device to shoot the screen of a second electronic device, the user can adjust the rotation angle of the first information displayed on the screen of the first electronic device relative to the screen of the second electronic device by changing the relative angle between the first electronic device and the second electronic device. It is understandable that the display angle of the first information on the second electronic device is proportional to the rotation angle of the first information displayed on the screen of the first electronic device relative to the screen of the second electronic device, because the rotation angle of the first information displayed on the screen of the first electronic device relative to the screen of the second electronic device is proportional to the relative angle between the first electronic device and the second electronic device. Therefore, the larger the first relative angle, the larger the corresponding first display angle.

[0017] It is understandable that by adjusting the relative angle between the first electronic device and the second electronic device, the user can adjust the display angle of the graphic element / image on the second electronic device accordingly, thereby improving the user experience.

[0018] In a possible implementation of the first aspect above, the first information includes a first graphic element in a first image, and the method further includes: after the first electronic device detects the first sharing operation, it extracts multiple graphic elements in the first image or a first area selected by the user in the first image; the first electronic device displays multiple graphic elements and sends the first graphic element selected by the user from the multiple graphic elements to the second electronic device.

[0019] It can be understood that after the first electronic device extracts multiple graphic elements, the user can select the graphic elements that he or she desires to transmit, which provides the user with many choices and improves the user's selectivity.

[0020] In a possible implementation of the first aspect above, the first information includes a first graphic element in the first image, and the method also includes: the first electronic device extracts a graphic element in the first image after detecting the first sharing operation; the first electronic device displays the extracted graphic element as the first graphic element, and sends it to the second electronic device.

[0021] It can be understood that after the first electronic device extracts a graphic element, it transmits it to the second electronic device, thereby improving the efficiency of the sharing operation.

[0022] In a possible implementation of the first aspect above, the first information is displayed in a first application interface of the second electronic device.

[0023] In a possible implementation of the first aspect above, it also includes: the first electronic device sends to the second electronic device first screen background image information of a first screen background image (i.e., the first screen background hereinafter) obtained by photographing the display screen of the second electronic device with the first location information; and the second electronic device determines the first display parameter based on the first screen background image information.

[0024] In a possible implementation of the first aspect above, the first screen background image information includes multiple first image feature points of the first screen background image (for example, the first feature points hereinafter); and the second electronic device determines the first display parameters based on the first screen background image information, including: the second electronic device determines the first display parameters based on the multiple first image feature points of the first screen background image, and multiple second image feature points (for example, the second feature points hereinafter) in the first screen image of the current display interface of the second electronic device corresponding to the multiple first image feature points.

[0025] It can be understood that the first screen image of the current display interface of the second electronic device is the second screen background below.

[0026] It can be understood that the first electronic device sends the image feature points as the screen background image information to the second electronic device. Since the image feature points are smaller than the original image data, transmitting the feature point information can effectively save the amount of transmitted data, thereby reducing transmission power consumption.

[0027] In a possible implementation of the first aspect above, the first display parameter includes a first display ratio, and the second electronic device determines the first display parameter based on multiple first image feature points of the first screen background image and multiple second image feature points corresponding to the multiple first image feature points in the first screen image of the current display interface of the second electronic device, including: obtaining a spatial transformation matrix according to the multiple first image feature points and the multiple second image feature points; determining, according to the spatial transformation matrix, a second center point and a second offset point corresponding to the first center point and the first offset point of the first screen background image from the first screen image of the current display interface of the second electronic device; calculating a first distance according to the coordinates of the first center point and the coordinates of the first offset point, and calculating a second distance according to the coordinates of the second center point and the coordinates of the second offset point; and obtaining the first display ratio according to the ratio between the first distance and the second distance.

[0028] In a possible implementation of the first aspect above, the first display parameter includes a first display angle, and the second electronic device determines the first display parameter based on multiple first image feature points of the first screen background image and multiple second image feature points corresponding to the multiple first image feature points in the first screen image of the current display interface of the second electronic device, including: obtaining a spatial transformation matrix according to the multiple first image feature points and the multiple second image feature points; determining, according to the spatial transformation matrix, a second center point and a second offset point corresponding to the first center point and the first offset point of the first screen background image from the first screen image of the current display interface of the second electronic device; calculating a first angle between a straight line where the first center point and the first offset point are located and a first coordinate axis in a coordinate system of the second electronic device, and calculating a second angle between a straight line where the second center point and the second offset point are located and the first coordinate axis in the coordinate system of the second electronic device; and obtaining the first display angle according to the difference between the first angle and the second angle.

[0029] In a second aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; one or more memories; one or more memories storing one or more programs, wherein when one or more programs are executed by one or more processors, the electronic device executes any display method as described in the first aspect and any one of the various implementations of the first aspect.

[0030] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the computer executes the display method as described in the first aspect and any one of the various implementations of the first aspect.

[0031] In a fourth aspect, a computer program product is provided, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes a display method as in the first aspect or any one of the possible implementations of the first aspect.

[0032] In a fifth aspect, a chip is provided. When the chip is running, the chip executes the display method of the first aspect or any one of the possible implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 According to some embodiments of the present application, a schematic diagram of a mobile phone 100 using a cloud 300 to share data with a computer 200 is shown;

[0034] Figure 2 According to some embodiments of the present application, a schematic diagram of a process in which a mobile phone 100 extracts a graphic element through a computer 200 and displays the graphic element at a corresponding position on the computer 200 is shown;

[0035] Figure 3A According to some embodiments of the present application, a schematic diagram of connecting a mobile phone 100 and a computer 200 is shown;

[0036] Figure 3B According to some embodiments of the present application, a schematic diagram of a process of extracting and displaying a graphic element by a mobile phone 100 is shown;

[0037] Figure 3C According to some embodiments of the present application, a schematic diagram of a computer 200 displaying graphic elements after a mobile phone 100 is rotated is shown;

[0038] Figure 4A According to some embodiments of the present application, another schematic diagram of a computer 200 displaying graphic elements after a mobile phone 100 is rotated is shown;

[0039] Figure 4B According to some embodiments of the present application, a schematic diagram of a first screen background B1 is shown;

[0040] Figure 4C According to some embodiments of the present application, a schematic diagram of a mobile phone screen W4 and a mobile phone screen W5 corresponding to a distance h1 between the mobile phone 100 and the screen of the computer 200, and a distance h2 between the mobile phone 100 and the screen of the computer 200 are shown when the relative angles between the mobile phone 100 and the screen of the computer 200 are the same;

[0041] Figure 5 According to some embodiments of the present application, a schematic diagram of a display method process is shown;

[0042] Fig. 6A According to some embodiments of the present application, a schematic diagram of a mobile phone 100 acquiring an image by shooting is shown;

[0043] Figure 6B According to some embodiments of the present application, a schematic diagram of a mobile phone 100 acquiring a foreground image through a gallery is shown;

[0044] Figure 6C According to some embodiments of the present application, a schematic diagram of transmitting a displayed graphic element by a mobile phone 100 to a computer 200 is shown;

[0045] Fig.6D According to some embodiments of the present application, a three-dimensional schematic diagram showing a computer 200 displayed on a screen of a mobile phone 100 is shown;

[0046] Fig. 6E According to some embodiments of the present application, a schematic diagram showing a correspondence relationship between pixel points on a first screen background of a mobile phone 100 and pixel points on a second screen background of a computer 200 is shown;

[0047] Fig. 6F According to some embodiments of the present application, a schematic diagram showing a comparison between a computer 200 displaying without post-processing optimization and after post-processing optimization is shown;

[0048] Fig. 7A According to some embodiments of the present application, a schematic diagram of a process of obtaining a display angle and a display ratio is shown;

[0049] Figure 7B According to some embodiments of the present application, a schematic diagram showing the correspondence between the center point and the offset point in the first screen background and the second screen background is shown;

[0050] Figure 8 According to some embodiments of the present application, a schematic diagram of the functional module structure of a mobile phone 100 is shown;

[0051] Fig. 9 According to some embodiments of the present application, a schematic diagram of the software system structure of a mobile phone 100 is shown;

[0052] Fig.10 According to some embodiments of the present application, a schematic diagram of the hardware structure of an electronic device is shown. DETAILED DESCRIPTION

[0053] The illustrative embodiments of the present application include, but are not limited to, a display method, a medium, and an electronic device.

[0054] The technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0055] As described in the background art, since the computer 200 can only display the graphics element at a fixed position on the screen, it is not possible to directly display the graphics element at a position the user wants to place, and the interactivity with the user is poor.

[0056] In some embodiments, after the mobile phone 100 obtains the image element that the user wants, the user can use the camera function of the mobile phone 100 to shoot the screen of the computer 200 while the screen showing the image element is displayed. At this time, the screen of the mobile phone 100 can show the specific position of the image element placed on the computer 200. Then the mobile phone 100 sends the image showing the specific position of the image element on the screen of the computer 200 to the computer 200, and at this time, the computer 200 can place it at the corresponding position.

[0057] Specifically, Figure 2 The figure shows a process of the mobile phone 100 extracting the image element through the computer 200 and displaying the image element on the corresponding position of the computer 200. In addition, the mobile phone 100 and the computer 200 establish a communication connection through wireless communication technology, for example, through Bluetooth, wireless network, etc., to perform data transmission. For example, Figure 2 As shown, the process may include the following steps:

[0058] S01: The mobile phone 100 sends the acquired foreground image to the computer 200, where the foreground image includes the graphics elements to be extracted.

[0059] It can be understood that an image includes a plurality of regions with significant features. For example, the region with significant features (ie, significant region) may be a specific person, object, text, etc. The region with significant features may be used as a primitive.

[0060] For example, Figure 3A As shown, the user wants to make a poster using the image processing software on the computer 200. At this time, there is already a "basket" on the poster. The user can use the material sharing software on the mobile phone 100 to obtain image materials. The computer 200 and the mobile phone 100 are connected through the network, and the material sharing software on the mobile phone 100 supports the transmission of image elements with the image processing software on the computer 200. The "shoot" button P1 in the material sharing software on the mobile phone 100 supports shooting to obtain the foreground image, and the "gallery" button P2 in the material sharing software on the mobile phone 100 supports obtaining the foreground image from the gallery. The user obtains as shown in FIG. Figure 3B After the foreground image shown on the mobile phone screen W1 is selected, the screen of the mobile phone 100 can be clicked, and the mobile phone 100 will display the foreground image as shown in FIG. Figure 3B The foreground image shown on the mobile phone screen W1 is sent to the computer 200.

[0061] It is understandable that, in addition to using the above-mentioned image processing software, the computer 200 and the mobile phone 100 can use any other application, such as a third-party application or an application supported by the system, and there is no limitation here.

[0062] S02: The server in the computer 200 extracts the image elements in the foreground image. For example, the image elements in the foreground image may be extracted to obtain the image element C1 of "apple" and the image element C2 of "bowl".

[0063] S03: The computer 200 sends the extracted graphic elements to the mobile phone 100, for example, the extracted “apple” graphic element C1 and “bowl” graphic element C2 are sent to the mobile phone 100.

[0064] S04: After receiving the graphic element, the mobile phone 100 may select a graphic element with a relatively large size by default and display it on the screen.

[0065] For example, in Figure 3BIn the example, the "apple" graphic element C1 is displayed on the mobile phone screen W2. At this time, the mobile phone 100 can shoot the screen of the computer 200 while displaying the graphic element. When the mobile phone 100 detects that the user clicks, or the mobile phone 100 automatically detects that it is pointing at the computer 200, the background image can be obtained. For example, the user uses the camera function of the mobile phone 100 to hold the mobile phone 100 displaying the "apple" graphic element and shoot at the shooting area sq1 of the computer 200, and obtains the following image: Figure 3B The background image shown on the mobile phone screen W3.

[0066] S05: The mobile phone 100 sends the background image to the computer 200.

[0067] S06: The server in computer 200 calculates the center position of the screen of computer 200 where the graphic element needs to be placed based on the background image. For example, based on the specific position of the center point Q1 of the "apple" graphic element C1 in the background image on the captured shooting area sq1, the server calculates the center position of the screen of computer 200 where the center point Q1 of the "apple" graphic element C1 needs to be placed.

[0068] S07: The image processing software of the computer 200 calls the software interface to place the image element at the calculated center position. For example, Figure 3B In the display screen 002A of the computer 200, the "apple" graphic element C1 is placed at the center position on the computer screen.

[0069] However, after receiving the background image, the computer 200 only calculates the center position of the image element displayed on the computer screen. At this time, the size or rotation angle of the image element in the background image relative to the shooting area is not considered during display, and the user interaction is not strong.

[0070] Specifically, the size of the image element in the background image relative to the shooting area will not be considered when the computer is displayed. It is understandable that, since the camera of the mobile phone 100 has a perspective function, the shooting area captured by the mobile phone 100 at a certain relative distance from the computer 200 is different from the actual shooting area. The relative distance can be understood as the distance from the center of gravity of the mobile phone 100 to the computer plane, where the computer plane is the plane where the computer 200 screen is located. For example, Figure 3B On the mobile phone screen W3 in the mobile phone 100, at this time, the shooting area displayed on the background image of the mobile phone 100 is different from the size of the actual shooting area sq1. Therefore, when the primitive is displayed on the shooting area sq1 in the computer 200 with the same size as in the background image, the effect required by the user in the background image cannot be achieved. For example, Figure 3B In the figure, the area of ​​the shooting area occupied by the “apple” graphic element displayed on the screen 002A displayed by the computer 200 is smaller than the area of ​​the shooting area occupied by the “apple” graphic element in the background image of the mobile phone 100.

[0071] Furthermore, the rotation angle of the image element relative to the captured area in the background image is not taken into account during display.

[0072] It can be understood that the image data captured by the mobile phone 100 is as follows: Figure 3C The XOY coordinate system shown in FIG. 1 (for example, the long frame of the mobile phone is the Y axis and the short frame of the mobile phone is the X axis) is used for coordinate representation, and the computer 200 is as follows Figure 3C The coordinates are expressed in an X'OY' coordinate system (for example, with the long computer frame as the X-axis and the short computer frame as the Y-axis).

[0073] Since the mobile phone 100 can be rotated by the user during the shooting process to adjust the specific position of the image elements on the screen of the mobile phone 100 on the screen of the computer 200 being photographed, there is a relative angle between the mobile phone 100 and the computer 200 at this time, wherein the relative angle can be the angle value between the central axis of the screen of the mobile phone 100 and the central axis of the screen of the computer 200, wherein when the mobile phone 100 and the computer 200 are placed upright along the direction of gravity, the central axes of the two are parallel.

[0074] When the mobile phone 100 and the computer 200 rotate relative to each other in space, the relative angle between the mobile phone 100 and the computer 200 is as follows: Figure 3C The relative angle between the central axis Z1 of the mobile phone 100 and the central axis Z2 of the computer 200 shown in . In the picture taken by the mobile phone 100, the primitive does not change in the XOY coordinate system, but it is rotated in the X'OY' coordinate system, and the computer 200 changes in the picture taken by the mobile phone 100 in the XOY coordinate system, but does not rotate in the X'OY' coordinate system, so in the picture taken by the mobile phone 100, the primitive is rotated relative to the screen of the computer 200. For example, taking the X'OY' coordinate system as the standard, as Figure 3C As shown, when the mobile phone 100 displays Figure 3C When the mobile phone screen W3 is displayed, the user rotates the mobile phone 100. At this time, since the specific position of the "apple" primitive on the mobile phone screen does not change, the "apple" primitive rotates with the rotation of the mobile phone. At this time, the "apple" primitive rotates in the X'OY' coordinate system, while the screen of the computer 200 in the shooting area captured by the mobile phone does not rotate in the X'OY' coordinate system, so the mobile phone 100 displays the following: Figure 3C In the mobile phone screen W4, the "apple" graphic element rotates relative to the photographed computer 200. When the user clicks, a background image such as the mobile phone screen W4 is obtained. It can be seen from the background image that the user wants the "apple" graphic element to be rotated and displayed on the screen of the photographed computer 200, but Figure 3CThe “apple” graphic element displayed in the photographing area sq2 on the display screen 002B of the computer 200 does not rotate.

[0075] Therefore, in order to solve the above problems, the present application proposes a display method. When there is a specific relative angle and a specific relative distance between a first electronic device (such as a mobile phone 100) and a second electronic device (such as a computer 200), the screen of the first electronic device displays a photographic picture of the specific placement position of the graphic element that the user wants on the screen of the second electronic device (such as a computer 200). At this time, the second electronic device can adjust the display angle and display ratio of the graphic element on the screen of the second electronic device according to the rotation angle of the graphic element in the photographic picture of the first electronic device relative to the screen of the second electronic device, and the relative size of the graphic element relative to the photographed screen of the second electronic device, according to the rotation angle and relative size relative to the screen of the second electronic device in the photographic picture, so that the placement of the graphic element displayed on the second electronic device achieves the effect of being consistent with the rotation angle and relative size of the graphic element in the photographic picture of the first electronic device relative to the screen area of ​​the second electronic device, thereby improving user interactivity.

[0076] The display angle of the graphic element on the screen of the second electronic device is the value of the angle between the central axis running through the center point of the graphic element on the screen of the second electronic device and the central axis of the screen of the second electronic device.

[0077] The display ratio of the graphic element placed on the screen of the second electronic device is the ratio between the size of the graphic element displayed on the second electronic device and the size of the graphic element displayed on the first electronic device.

[0078] For example, Figure 4A As shown, for the mobile phone screen W4 of the mobile phone 100, the "Apple" graphic element on the display screen 002C of the computer 200 is displayed according to the display angle corresponding to the rotation angle in the mobile phone screen W4 of the mobile phone 100, and is enlarged according to the display ratio corresponding to the relative size in the display screen, so that the area occupied in the shooting area is similar to the area occupied in the mobile phone screen W4 of the mobile phone 100, thereby improving the user's interactivity. At this time, the display angle is as follows Figure 4A The angle between the central axis Z3 passing through the center point Q1 of the "apple" primitive and the central axis Z2 of the screen of the computer 200 is shown in FIG. When the plane where the screen of the computer 200 is located is parallel to the plane where the screen of the mobile phone 100 is located, Z3 is parallel to the central axis Z1 of the mobile phone 100.

[0079] Specifically, the computer 200 can calculate the rotation angle and relative size of the captured shooting area relative to the actual display screen of the computer 200 in the X'OY' coordinate system based on the relevant data of the first screen background (i.e., the screen corresponding to the shooting area) in the XOY coordinate system of the mobile phone 100 and the relevant data of the actual display screen in the current X'OY' coordinate system, so as to obtain the display angle and display ratio that need to be adjusted when placing the graphic element.

[0080] For example, in some embodiments, the mobile phone 100 can send the feature points of the first screen background in the XOY coordinate system to the computer 200, and the computer 200 calculates the display angle and display ratio corresponding to the placement of the graphic element according to the feature points of the first screen background. Among them, the feature points can reflect the significant information of the first screen background, and the values ​​of the feature points may include pixel values, coordinates, scales (Gaussian parameters), main directions, and directional gradient descriptors. For example, Figure 4B Shown with Figure 4A The first screen background B1 corresponding to the mobile phone screen W4 in the image is extracted. At this time, the feature points of the first screen background B1 can be extracted and sent to the computer 200. For example, there are 1000 feature points of the first screen background, among which one feature point can be the pixel point information representing the "basket" located at the edge of the first screen background, and one feature point can be the pixel point information representing the center of the first screen background, etc.

[0081] It can be understood that the feature point information of the first background image is smaller than the original image data to be sent, so transmitting the feature point information can effectively save the amount of sent data, thereby reducing transmission power consumption.

[0082] It is understandable that in some embodiments, when the user uses the screen of the mobile phone 100 to shoot the screen of the computer 200, the relative size of the image element in the screen display and the screen of the computer 200 can be adjusted by zooming in / out on the mobile phone 100. Specifically, when the user moves the screen of the mobile phone 100 away from the screen of the computer 200, the screen of the computer 200 will become smaller in the picture displayed on the screen of the mobile phone 100, and since the size of the image element remains unchanged, the image element will become larger relative to the screen of the computer 200; for another example, when the user moves the screen of the mobile phone 100 close to the screen of the computer 200, the screen of the computer 200 will become larger in the picture displayed on the screen of the mobile phone 100, and since the size of the image element remains unchanged, the image element will become smaller relative to the screen of the computer 200.

[0083] For example, suppose Figure 4C The shooting distance corresponding to the mobile phone screen W5 is h2. Figure 4CThe figure shows the mobile phone screens at different relative distances (the distance h1 and the distance h2 in the figure are specific different relative distances) when the relative angle between the mobile phone 100 and the computer 200 in space is consistent. The mobile phone screens W4 and W5 corresponding to the distance h1 between the mobile phone 100 and the screen of the computer 200, and the distance h2 between the mobile phone 100 and the screen of the computer 200. Since the distance h1 is greater than the distance h2, the shooting area sq2 photographed at the distance h1 is larger than the shooting area sq3 photographed at the distance h2. Therefore, the size of the graphic element "apple" C1 relative to the screen of the computer 200 is larger at the distance h1 than at the distance h2. Therefore, Figure 4C As shown, when the screen of computer 200 displays the "apple" graphic element, the "apple" on the screen of computer 200 corresponding to the mobile phone screen W4 is larger than the "apple" on the screen of computer 200 corresponding to the mobile phone screen W5.

[0084] It can be understood that the display method mentioned in the present application can be applied not only to the scenario mentioned above where posters need to be made, but also to any other scenario where other applications need to add image materials, without limitation here.

[0085] It can be understood that in the display method mentioned in the present application, in addition to transferring graphic elements to the computer 200, the mobile phone 100 can also transfer the entire image to the computer 200. At this time, the mobile phone 100 does not need to perform graphic element operations, and directly displays the image captured or obtained from the gallery at a preset position on the screen of the mobile phone 100 according to a certain size. Then, the screen of the mobile phone 100 is photographed while displaying the image, so that the image on the computer 200 is displayed according to the corresponding display ratio and display angle.

[0086] It can be understood that the first electronic device and the second electronic device mentioned in the present application can be any electronic device with a screen display function, including but not limited to mobile phones, tablet computers, vehicle-mounted equipment, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), servers, etc. The embodiments of the present invention do not impose any restrictions on the first electronic device and the second electronic device.

[0087] It can be understood that it has been mentioned above that the screen image captured by the mobile phone without superimposed graphics is called the "first screen background". At this time, for the convenience of explanation, the actual screen image of the computer 200 is called the "second screen background".

[0088] The display process is described in detail below by taking the first electronic device as a mobile phone 100 and the second electronic device as a computer 200 as an example.

[0089] Figure 5 According to some embodiments of the present application, a schematic diagram of a process is shown in which, after a mobile phone 100 obtains an image element and uses a camera to shoot the screen of a computer 200, the image element can be displayed on the screen of the computer 200 according to a display angle and a display ratio. The specific process includes:

[0090] S501, the mobile phone 100 generates metadata.

[0091] In some embodiments, the mobile phone 100 can obtain images through other channels such as a gallery or shooting, call a cutout algorithm to extract a significant area, thereby generating metadata, and display the generated metadata at a preset position of the computer 200, that is, the center point of the primitive corresponds to the preset center position. In addition, the mobile phone 100 sends the primitive displayed at the preset position of the computer 200 to the computer 200.

[0092] In some implementations, when there are multiple salient areas on an image, the mobile phone 100 can extract multiple image primitives corresponding to the multiple salient areas in the image, or extract multiple image primitives in the area selected by the user in the image, generate multiple image primitives, and display the generated multiple image primitives at a preset position of the computer 200. In addition, the mobile phone 100 will send the multiple primitives displayed at the preset position of the computer 200 to the computer 200.

[0093] In other implementations, when there are multiple significant areas on an image, the mobile phone 100 can extract multiple image elements, generate multiple image element data, and then display them on the screen for the user to select the desired image element, and then redisplay the image element selected by the user separately at the preset position of the computer 200. In addition, the mobile phone 100 will send the separately redisplayed image element at the preset position of the computer 200 to the computer 200.

[0094] In addition, in some implementations, when there are multiple significant areas on an image, the mobile phone 100 can extract the most significant area by default, generate one metadata, and then display the generated one metadata at a preset position of the computer 200. In addition, the mobile phone 100 sends the metadata displayed at the preset position of the computer 200 to the computer 200.

[0095] For example, Fig. 6A The mobile phone 100 shown in FIG. 1 is a schematic diagram of capturing a foreground image and displaying a primitive. Fig. 6AIn the mobile phone screen W11 shown in FIG. 1 , the user selects the “shoot button” P1 to turn on the camera function of the mobile phone 100, and the user uses the mobile phone 100 to shoot the apple and the bowl on the table. When the user clicks the screen of the mobile phone 100 to confirm shooting, Fig. 6A The mobile phone screen W12 shown in FIG. 1 is a screen shot by the user after clicking the mobile phone screen. The image corresponding to the shot is the foreground image. At this time, the mobile phone 100 can use the cutout algorithm to extract the primitives. Assuming that the default setting extracts the primitives of the most prominent area, the apple area in the mobile phone screen W12 can be extracted to generate the "apple" primitive C1. Fig. 6A In the mobile phone screen W13 shown in FIG. 1 , the generated “apple” graphic element C1 is displayed at the center of the screen of the mobile phone 100. At this time, the center point of the “apple” graphic element C1 coincides with the center position. Figure 6C As shown, the mobile phone 100 will be connected with Fig. 6A The "apple" graphic element C1 corresponding to the mobile phone screen W13 shown in the figure is sent to the computer 200 as the first screen foreground, so that the computer 200 can subsequently adjust the "apple" graphic element C1 and display it.

[0096] For example, Figure 6B FIG. 1 shows a schematic diagram of a mobile phone 100 acquiring a foreground image through a gallery. Figure 6B In the mobile phone screen W21 shown in FIG. 1 , the user clicks the “Gallery” button P2, and the mobile phone 100 can enter the gallery. Figure 6B In the mobile phone screen 22 in FIG. 6, 6 images are displayed. The user clicks an image in the upper left corner. At this time, the image clicked by the user is displayed on the mobile phone as the foreground image. For example, Figure 6B In the mobile phone screen 23, the mobile phone 100 displays the upper left corner image clicked by the user on the mobile phone screen W22 as the foreground image. It can be understood that after the mobile phone 100 obtains the foreground image, the process of extracting and displaying the primitives is substantially the same as the process of extracting and displaying the primitives after the foreground image is obtained by shooting, and will not be repeated here.

[0097] S502, the mobile phone 100 takes a photo of the screen of the computer 200 at a relative position expected by the user, wherein the relative position includes a relative angle and a relative distance.

[0098] It is understandable that the mobile phone 100 can shoot the screen of the computer 200 while displaying the screen of the graphic element, and then adjust the relative angle and relative distance between the mobile phone 100 and the computer 200 (for example, rotation, zooming in / out, etc.) during the shooting process, so as to adjust the specific center position, relative size and rotation angle of the graphic element on the screen of the mobile phone 100 relative to the captured screen of the computer 200, and then obtain the desired graphic element placement effect. After detecting that the current shooting effect is the effect expected by the user, the mobile phone 100 sends the first screen background related data (for example, the feature points of the first screen background obtained by the feature extraction algorithm) to the computer 200.

[0099] Specifically, in some embodiments, the camera function of the mobile phone 100 is used to shoot the computer 200, and at this time, the mobile phone 100 will obtain the first screen background. Fig.6D The figure shows the corresponding situation of the screen of the computer 200 displayed on the screen of the mobile phone 100. When the user uses the mobile phone 100 to shoot the screen of the computer 200 (i.e., the second screen background), the three-dimensional space content will be mapped to the two-dimensional plane, and the picture displayed on the screen of the mobile phone 100 will correspond to the partial picture content of the screen of the computer 200. Specifically, there is a spatial mapping relationship between the partial content on the first screen background and the partial content on the second screen background. As shown in the figure, the pixel points rt1, rt2, and rt3 on the screen of the computer 200 (the second screen background) correspond to the pixel points r1, r2, and r3 on the screen of the mobile phone 100 (the first screen background), respectively.

[0100] In order to obtain the desired effect of placing graphic elements, when the mobile phone 100 is shooting, the graphic elements will be displayed on the screen of the mobile phone 100, and the user uses the mobile phone 100 displaying the graphic elements to shoot the computer 200. At this time, the graphic elements will be superimposed on the photographed computer 200.

[0101] It is understandable that, in the process of shooting with the mobile phone 100 against the computer 200, the user can adjust the relative distance and relative angle of the mobile phone 100 (such as rotation, zooming in / out, tilting, etc.) so that the image element on the screen of the mobile phone 100 is placed at the desired position on the screen of the computer 200 being photographed. When the user performs the preset "end shooting" operation, the mobile phone 100 detects the user's "end shooting" operation instruction, and extracts feature points from the image data of the first screen background in the displayed image using a feature extraction algorithm, and sends the extracted first feature points to the computer 200, wherein the first feature points can reflect the content information of the first screen background.

[0102] It is understandable that the first feature point information is smaller than the original image data, which effectively saves the amount of data sent. Among them, the feature extraction algorithm can be a variety of scale-invariant feature transform (SIFT), local binary patterns (LBP) algorithm, feature extraction algorithm (histogram of oriented gradient, HOG), etc. It is understandable that the value of the feature point can include pixel value, coordinates, scale (Gaussian parameter), main direction, and directional gradient descriptor, etc.

[0103] For example, Figure 4A In the mobile phone screen W3 to the mobile phone screen W4, the user rotates the mobile phone 100, and corresponding to the mobile phone screen W4, the user clicks the screen to perform the preset "end shooting" operation. After the mobile phone 100 detects the user's "end shooting" operation instruction, the mobile phone 100 will Figure 4B The first screen background B1 shown is subjected to shift feature point extraction to obtain a first feature point, and the first feature point is sent to the computer 200. The first feature point includes significant information of the first screen background B1, such as the size of the first screen background B1, and information corresponding to the screen content (such as a basket, etc.) of the computer 200. It can be understood that the feature point data of the first screen background B1 of the mobile phone 100 is represented in accordance with the data representation method of the coordinate system of the mobile phone 100.

[0104] It can be understood that the above predetermined operation realizes the shooting process. In addition to the user clicking the screen, physical buttons (such as the back button, volume button, power button) can also be used to realize the shooting process. Long press can also be changed to single click, double click, press or slide, etc. The specific predetermined operation is not repeated here.

[0105] It is understandable that in other embodiments, when the mobile phone 100 is shooting at the computer 200, the user may adjust the relative distance and relative angle between the mobile phone 100 and the computer 200. The user does not need to perform a predetermined operation at this time. After the mobile phone 100 automatically detects that it is aimed at the computer 200 (for example, the user remains stationary at the current position for more than a preset number of seconds), the computer 200 is shot.

[0106] S503, the computer 200 performs screen background image matching.

[0107] It is understandable that there will be a large gap between the pixel data features of the first screen background and the screen image content of the computer 200 obtained by shooting the computer 200 with the mobile phone 100 and the actual display image of the second screen background on the computer 200. At this time, it is necessary to perform feature matching on the first screen background and the second screen background shot by the mobile phone 100, and find pixel points in the second screen background that are relatively similar to the first feature points of the first screen background, so as to obtain feature points (for the convenience of explanation, referred to as "second feature points") used to represent the pixel points that are relatively similar to the features found. A set of registration points is obtained based on the first feature points and the second feature points, and there is a one-to-one correspondence between the first feature points and the second feature points in the registration point set.

[0108] In some embodiments, a matching algorithm may be used, for example, a nearest neighbor (k-nearest neighbor, kNN) to screen the first feature point of the first screen background and each pixel point in the second screen background to find pixel points with similar features, thereby obtaining a second feature point. The first feature point of the first screen background and the second feature point found in the second screen background are used as a set of registration points. For example, the first feature point of the first screen background is: ST{ST1, ST2, ST3, ..., STN}; the second feature point of the second screen background is: DT{DT1, DT2, DT3, ..., DTN}.

[0109] For example, Fig. 6E FIG. 1 shows the pixel points S1, S2 and S3 corresponding to the three feature points ST1, ST2 and ST3 in the first screen background displayed on the display screen B2 of the mobile phone 100. Feature point ST1 corresponds to the pixel point S1 on the upper left edge of the basket, feature point ST2 corresponds to the pixel point S2 on the lower edge of the basket, and feature point ST3 corresponds to a pixel point S3 on the upper right edge of the basket. And, Fig. 6E Also shown in the figure are the pixel points corresponding to the feature points DT1, DT2, and DT3 that match the feature points ST1, ST2, and ST3 in the second screen background 002D of the computer 200. Feature point DT1 corresponds to pixel point D1 on the upper left edge of the basket, feature point DT2 corresponds to a pixel point D2 on the lower edge of the basket, and feature point DT3 corresponds to pixel point D3 on the upper right edge of the basket. It can be understood that since the features of ST1, ST2, and ST3 correspond to the features of DT1, DT2, and DT3, the three pixel points S1, S2, and S3 in the first screen background correspond one-to-one to the three pixel points D1, D2, and D3 in the second screen background.

[0110] S504, the computer 200 determines whether the matching is successful.

[0111] It is understandable that when the mobile phone 100 is shooting at the computer 200, the background of the first screen captured may not necessarily contain feature points that match the pixel point data in the background of the second screen.

[0112] In some embodiments, when the computer 200 matches the first feature point of the first screen background with the pixel point data of the second screen background, if no pixel point with similar features is found in the pixel point data of the second screen background, the computer 200 determines that the match is unsuccessful, and then enters S502, requiring the mobile phone 100 to re-interact between screens to re-obtain the relevant data of the first screen background that can be used for matching. When the computer 200 finds a pixel point with similar features in the pixel point data of the second screen background and determines that the match is successful, it enters S505, and the computer 200 adjusts the primitives for display.

[0113] S505, the computer 200 displays the graphic element.

[0114] In some embodiments, the spatial mapping relationship between the first screen background and the second screen background is obtained according to the set of registration points. It is understandable that the spatial mapping relationship corresponds to the relative position between the mobile phone 100 and the computer 200. It is understandable that, according to the obtained spatial mapping relationship, the position corresponding to any pixel point on the screen of the computer 200 photographed on the mobile phone 100 on the actual screen of the computer 200 can be obtained. At this time, according to the spatial mapping relationship, the corresponding center position of the center point of the primitive in the second screen background, the display ratio and display angle of the primitive when displayed on the screen of the computer 200 can be obtained. Then, the primitive is displayed at the corresponding center position according to the display ratio and display angle. Among them, the specific process of obtaining the display ratio and display angle required for the primitive according to the spatial mapping relationship is described in Figure 7 below, and will not be repeated here.

[0115] For the convenience of explanation, the corresponding point of the center point of the graphic element in the first screen background is called the "first center point", and the corresponding point of the center point of the graphic element in the second screen background is called the "second center point".

[0116] Specifically, after the computer 200 obtains the first center point of the first screen background captured by the mobile phone 100, the spatial mapping relationship is used to find the corresponding second center position of the first center point on the first screen background on the actual screen of the computer 200 (i.e., the second screen background). After the received graphics element is displayed on the screen of the computer 200 according to the display angle and display ratio obtained according to the spatial mapping relationship, the graphics element is placed at the second center position in an Alpha blending and superposition manner, and edge filtering optimization is performed at the same time to obtain the final graphics element display effect.

[0117] For example, assuming that the first center position of the first center point of the "apple" graphic element on the mobile phone 100 is (a1, b1) in the XOY coordinate system, at this time, according to the spatial mapping relationship, the second center position of the first center point in the X'OY' coordinate system of the computer 200 screen is (a1', b1'), the display angle of the graphic element is θ1, and the display ratio is k1. The "apple" graphic element C1 received from the mobile phone 100 is displayed at a display angle of θ1 and scaled according to a display ratio of k1, and is placed at the second center position (a1', b1') through Alpha blending and overlay technology.

[0118] It can be understood that, for example, as shown in 6F, after the Alpha blending superposition technology, Fig. 6F In (1), the “apple” primitive C1 will have some burrs and distortions, which can be optimized using edge filtering technology, such as Fig. 6F As shown in (2), a smooth “apple” primitive C1 is obtained.

[0119] It can be understood that the execution order of the above steps S501 to S505 is only an illustration. In other embodiments, other execution orders may be adopted, and some steps may be split or combined, which is not limited here.

[0120] It can be understood that the present application obtains image generation primitives through the first electronic device (such as mobile phone 100), and through convenient screen interaction, the primitives can be accurately placed on the screen of the second electronic device (such as computer 200) according to the specific center position, display ratio, and display angle, and the interaction process is simple. Since the first electronic device only needs to transmit the primitive data and the first screen background related data (for example, the first feature point of the first screen background) to the second electronic device, the interaction process between devices is simplified and the data transmission volume between devices is optimized.

[0121] Fig. 7A According to some embodiments of the present application, a schematic diagram of a specific process of obtaining the display ratio and display angle that need to be adjusted when placing a graphic element on the screen of the computer 200 according to the spatial mapping relationship between the first screen background and the second screen background is shown. Specifically, Fig. 7A As shown, the specific process may include:

[0122] S701: Obtain a spatial mapping relationship between a first screen background and a second screen background according to a set of registration points.

[0123] In some embodiments, in the registration point sets of the first screen background and the second screen background, a pair of registration points with a high degree of similarity is selected to construct a spatial transformation matrix (e.g., a homography matrix) of the perspective transformation, and the spatial transformation matrix is ​​used as a spatial mapping relationship. It can be understood that the two planes of the first screen background and the second screen background are registered in space to find a spatial mapping relationship, which is related to spatial attributes such as the relative distance and relative angle between the mobile phone 100 and the computer 200 in space.

[0124] S702, selecting a first center point St1 and a first offset point St2 from the first background screen, and calculating the corresponding second center point Dt1 and second offset point Dt2 in the second background screen through the spatial mapping relationship between the first screen background and the second screen background.

[0125] It can be understood that the first offset point is any pixel point in the first background screen corresponding to the captured screen of the computer 200. For example, it can be any pixel point on the screen of the computer 200 captured on the first background screen that is superimposed by the apple.

[0126] In some embodiments, a first center point St1 and a second center point Dt1 and a second offset point Dt2 corresponding to a first offset point St2 in the second background screen are obtained according to a spatial transformation matrix (eg, a homography matrix).

[0127] For example, Figure 7B , a first center point St1, and a first offset point St2 corresponding to a second center point Dt1 and a second offset point Dt2 in the second background screen are shown.

[0128] S703, calculating the display ratio and display angle corresponding to the graphic element according to the first center point St1, the first offset point St2, the second center point Dt1 and the second offset point Dt2.

[0129] In some embodiments, for the display ratio required for the graphic element: the coordinates (st1_x, st1_y) and (st2_x, st2_y) of the first center point St1 and the first offset point St2 are obtained, and the Euclidean distance d1 between the first center point St1 and the first offset point St2 is calculated based on the coordinates; the coordinates (dt1_x, dt1_y) and (dt2_x, dt2_y) of the second center point Dt1 and the second offset point Dt2 are obtained, and the Euclidean distance d2 between the second center point Dt1 and the second offset point Dt2 is calculated. It can be understood that the ratio of d2 to d1 is the ratio of the size of the graphic element displayed on the screen by the computer 200 to the size of the graphic element received by the computer 200. The ratio of d2 to d1 will be used as the display ratio of the graphic element sharing.

[0130] In some embodiments, for the display angle required by the graphic element: obtain the coordinates (st1_x, st1_y) and (st2_x, st2_y) of the first center point St1 and the first offset point St2, and calculate the angle β1 between the straight line where the first center point St1 and the first offset point St2 are located and OY according to the coordinates; obtain the coordinates (dt1_x, dt1_y) and (dt2_x, dt2_y) of the second center point Dt1 and the second offset point Dt2, and calculate the angle β2 between the straight line where the second center point Dt1 and the second offset point Dt2 are located and OY' according to the coordinates, and the difference between the two angles |β1-β2| is used as the display angle shared by the graphic element.

[0131] For example, Figure 7B In the figure, when the computer 200 receives the metadata, the OY axis in the coordinate system XOY of the mobile phone 100 is parallel to the OY' in the coordinate system X'OY' of the computer 200. At this time, the angle between the straight line where the first center point St1 and the first offset point St2 are located and OY is calculated for the interface of the mobile phone 100 to obtain angle 1, and the angle between the straight line where the second center point Dt1 and the second offset point Dt2 are located and OY' is calculated for the interface of the computer 200 to obtain angle 2, and angle 1-angle 2 is subtracted to obtain the display angle.

[0132] It can be understood that the execution order of the above steps S701 to step 703 is only an illustration. In other embodiments, other execution orders may be adopted, and some steps may be split or combined, which is not limited here.

[0133] Figure 8 According to some embodiments of the present application, a schematic diagram of the structural modules of a mobile phone 100 and a computer 200 is shown, which enables the computer 200 to display adjusted graphic elements on the screen. Specifically, Figure 8 As shown, the mobile phone 100 includes an application 801A and a primitive calculation unit 802A. The computer 200 includes an application 801B and a primitive calculation unit 802B. The primitive calculation unit 802A cooperates with the primitive calculation unit 802B to implement the above Figure 5 The display method shown.

[0134] Specifically, the application 801A in the mobile phone 100 has an application program for displaying graphics elements and camera functions, for example, Figure 3A The application 801B in the computer 200 is an application program capable of displaying received graphics elements, for example, Figure 3A It is understandable that the application 801A in the mobile phone 100 and the application 801B in the computer 200 can respectively call the primitive calculation unit in each device, so that the computer 200 uses the mobile phone 100 to obtain the primitives to be displayed and display the primitives at the corresponding positions.

[0135] The primitive calculation unit 802A includes a connection subunit 8021A, a primitive generation subunit 8022A, and an inter-screen interaction subunit 8023A.

[0136] The primitive calculation unit 802B includes a connection subunit 8021B, a matching subunit 8022B, a primitive display subunit 8023B, and a data storage subunit 8024B.

[0137] The connection subunit 8021A and the connection subunit 8021B perform service initialization and device connection. The primitive generation subunit 8022A is used to generate primitive metadata. For example, the above Figure 5 The specific contents described in S501 in the above description will not be repeated here. At this time, the primitive generation subunit 8022A will send the primitive data to the computer 200, and the data storage subunit 8024B in the computer 200 will store the received primitive data.

[0138] The screen interaction subunit 8023A is used to display the image element on the screen, and use the camera function to shoot the computer 200 to obtain the first screen background, wherein the image element on the screen of the mobile phone 100 will be superimposed on the first screen background, and the acquired first screen background related data will be sent to the computer 200, and the data storage subunit 8024B in the computer 200 will store the received first screen background related data. For example, the above can be executed Figure 5 The specific contents described in S502 are not repeated here.

[0139] The matching subunit 8022B is used to match the stored first screen background related data with the second screen background related data to determine whether the match is successful. Figure 5 The specific contents described in S503 and S504 are not repeated here.

[0140] The graphic element display subunit 8023B is used to obtain the spatial mapping relationship between the first screen background and the second screen background, and based on the spatial mapping relationship, obtain the display angle, display ratio and corresponding second center position corresponding to the graphic element, and adjust the graphic element for display. Figure 5 The specific contents described in S505 will not be repeated here.

[0141] Understandably, Figure 8 The structural modules of the mobile phone 100 and the computer 200 shown are only examples. In other embodiments, other module structures may be used, which is not limited in the embodiments of the present application.

[0142] It can be understood that the mobile phone 100 and the computer 200 can also provide image materials to the computer 200, and the required graphics elements are displayed on the screen of the mobile phone 100. In this case, the mobile phone 100 also includes a matching subunit 8022B and a graphics element display subunit 8023B; correspondingly, the computer 200 also includes a graphics element generation subunit 8022A and a screen interaction subunit 8023A, which are not required here.

[0143] It is understandable that the applications supporting image element display in the computer 200 and the applications supporting image element acquisition and image element transmission on the mobile phone 100 mentioned in this application are not only the image processing software and material sharing software mentioned above, but also any other applications supporting this function, such as office applications, social software applications, specifically memos, notes, instant messaging software, etc., all of which are within the protection scope of this application. And, it is understandable that, Figure 8 The graphic primitive calculation unit in the mobile phone 100 and the computer 200 may be included in the application program, or may be a functional unit provided by the system in the mobile phone 100 and the computer 200. When the application programs in the mobile phone 100 and the computer 200 need to use the graphic primitive calculation functional unit, the application programs in the mobile phone 100 and the computer 200 may respectively call the corresponding functional module from the system to cooperate in realizing the graphic primitive display.

[0144] It is understandable that when the user shares image elements on the same display screen of the image processing software on the computer 200 for multiple times, the background of the second screen on the computer 200 will not change.

[0145] It can be understood that there is no strict master-slave role relationship between the first electronic device (such as mobile phone 100) and the second electronic device (such as computer 200), which facilitates flexible image sharing services between devices (i.e., the image sent by the first electronic device is displayed on the screen of the second electronic device).

[0146] further, Fig. 9 According to some embodiments of the present application, a software structure diagram of a mobile phone 100 is shown. Fig. 9 As shown, the software structure of the mobile phone 100 is a multi-layer structure, which includes, from top to bottom, an application layer 91, an application framework layer 92, a system library 93 and a kernel layer 94.

[0147] The application layer 91 may include a series of application packages. Fig. 9 As shown in FIG, the application layer 91 includes applications such as Bluetooth 910, music 911, gallery 912, camera 913, and material sharing 914. In some embodiments of the present application, material sharing 914 can provide image primitives for image materials that users want to display on the computer 200.

[0148] The application framework layer 92 provides an application programming interface (API) and a programming framework for the application of the application layer 91, and the application framework layer includes some predefined functions. The application framework layer 92 includes a notification manager 920, a view system 921, a content provider 922, a primitive calculation 923, a resource manager 924, etc. Among them, in some embodiments of the present application, the primitive calculation 923 is used to provide the material sharing 914 with an application programming interface and a programming framework for calling primitive-related operations in the system library.

[0149] The system library 93 is the core of the software system of the mobile phone 100, and can provide services to the applications of the application layer 91 through the application framework layer 92. The system service layer 93 includes a media library 930, a graphics processor 931, a surface management library 932, a signal processing library 933, a primitive calculation library 934, etc. Among them, in some embodiments of the present application, the primitive calculation library 934 can receive instructions from the primitive calculation 923 and perform primitive processing related operations. For example, the primitive calculation library 934 includes Figure 8 The middle primitive calculation unit 802A performs operations such as primitive generation and interaction between screens.

[0150] The kernel layer 94 includes a display driver 940, a camera driver 941, a network card driver 942, an audio driver 943, and a sensor driver 944. In some embodiments of the present application, the camera driver 941 can respond to the camera function and drive the camera to shoot.

[0151] Understandably, Fig. 9 The software structure of the mobile phone 100 is only an example. In other embodiments, the mobile phone 100 may also adopt other forms of software structures, such as Android TM 、iOS TM 、HarmonyOS TM 、Tizen TM The present application is not limited to the embodiments.

[0152] Fig.10 According to an embodiment of the present application, a schematic diagram of the hardware structure of a mobile phone 100 is shown.

[0153] exist Fig.10 In the drawings, similar parts have the same reference numerals. Fig.10As shown, the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0154] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0155] The processor 110 may include one or more processing units, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a microprocessor (MCU), an artificial intelligence (AI) processor, or a processing module or processing circuit of a programmable logic device (FPGA). Different processing units may be independent devices or integrated into one or more processors. In some embodiments, the processor 110 may perform the above Figure 5 Operations in a display method related to mobile phone 100.

[0156] The memory can be used to store data, software programs and modules, and can be a volatile memory (volatile memory), such as a random-access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, or it can also be a removable storage medium, such as a secure digital (SD) memory card.

[0157] The charging management module 140 may include a power source, a power management component, etc. The power source may be a battery. The power management component is used to manage the charging of the power source and the power supply of the power source to other modules. The charging management module is used to receive charging input from the charger; the power management module is used to connect the power source, the charging management module and the processor 110.

[0158] The mobile communication module 150 may include, but is not limited to, an antenna, a power amplifier, a filter, a low noise amplifier (LNA), etc. The mobile communication module 150 may provide a solution for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the mobile phone 100. The wireless communication module 160 may include an antenna, and transmit and receive electromagnetic waves via the antenna.

[0159] The mobile phone 100 can communicate with the network and other devices through wireless communication technology.

[0160] In some embodiments, the mobile communication module 150 and the wireless communication module 160 of the mobile phone 100 may also be located in the same module.

[0161] The camera 193 is used to capture still images or videos. The optical image generated by the lens is projected onto the photosensitive element. The photosensitive element converts the optical signal into an electrical signal, which is then passed to the image signal processor (ISP) to be converted into a digital image signal. The mobile phone 100 can realize the shooting function through the ISP, the camera 193, the video codec, the graphic processing unit (GPU), the display screen, and the application processor.

[0162] The display screen includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini LED, Micro LED, Micro OLED, quantum dot light-emitting diodes (QLED), etc.

[0163] The audio module 170 can convert digital audio information into analog audio signal output, or convert analog audio input into digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0164] The interface module includes an external memory interface, a universal serial bus (USB) interface, and a subscriber identification module (SIM) card interface. The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 through the external memory interface to implement a data storage function. The universal serial bus interface is used for the mobile phone 100 to communicate with other mobile phones. The subscriber identification module card interface is used to communicate with a SIM card installed in the mobile phone 100, for example, to read a phone number stored in the SIM card, or to write a phone number into the SIM card.

[0165] In some embodiments, the mobile phone 100 further includes buttons, motors, and indicators, etc. Among them, the buttons may include a volume button, a power on / off button, etc. The motor is used to make the mobile phone 100 produce a vibration effect. The indicator may include a laser indicator, a radio frequency indicator, an LED indicator, etc.

[0166] It is understandable that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100 .

[0167] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction or program code in the form of software. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor to be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0168] According to the method provided in the embodiments of the present application, the present application also provides a computer program product, which includes: computer program code, when the computer program code is executed on a computer, the computer implements the steps performed by the electronic device in any one of the above embodiments.

[0169] According to the method provided in the embodiments of the present application, the present application also provides a computer-readable medium, which stores program code. When the program code runs on a computer, the computer implements the steps performed by the electronic device in any one of the above embodiments.

[0170] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0171] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0172] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0173] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier wave, infrared signal digital signal, etc.) using the Internet in an electrical, optical, acoustic or other form of propagation signal. Therefore, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0174] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0175] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0176] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0177] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A display method, characterized in that: include: The first electronic device detects a first sharing operation of a user to share first information with a second electronic device, and the first electronic device captures a display screen of the second electronic device with first location information; The second electronic device displays the first information using a first display parameter; The first electronic device detects a second sharing operation of a user to share second information with a second electronic device, and the first electronic device captures a display screen of the second electronic device with second location information; The second electronic device displays the second information using a second display parameter; The first position information and the second position information include at least one of a relative distance and a relative angle between the first electronic device and the second electronic device, and the first display parameter and the second display parameter include at least one of a display ratio and a display angle.

2. The method according to claim 1, characterized in that The first information and the second information are images or picture elements in the images.

3. The method according to claim 2, characterized in that The display ratio is a ratio between a size of the first information displayed on the second electronic device and a size of the first information displayed on the first electronic device.

4. The method according to claim 3, characterized in that The first position information includes a first relative distance, and the first display parameter includes a first display ratio; The second position information includes a second relative distance, and the second display parameter includes a second display ratio, wherein The first relative distance is smaller than the second relative distance, and the first display ratio is smaller than the second display ratio.

5. The method according to claim 1, characterized in that Also includes: The first position information includes a first relative angle, and the first display parameter includes a first display angle; The second position information includes a second relative angle, the second display parameter includes a second display angle, Among them, the first relative angle is smaller than the second relative angle, and the first display angle is smaller than the second display angle.

6. The method according to claim 1, characterized in that The first information includes a first picture element in the first image, and the method further includes: After detecting the first sharing operation, the first electronic device extracts the first image or a plurality of graphic elements in a first area selected by a user in the first image; The first electronic device displays the plurality of graphic elements, and transmits a first graphic element selected by a user from the plurality of graphic elements to the second electronic device.

7. The method according to claim 1, characterized in that The first information includes a first picture element in the first image, and the method further includes: After detecting the first sharing operation, the first electronic device extracts a graphic element from the first image; The first electronic device displays the extracted graphic element as a first graphic element, and sends it to the second electronic device.

8. The method according to claim 1, characterized in that The first information is displayed in a first application interface of the second electronic device.

9. The method according to claim 1, characterized in that: The method further comprises: The first electronic device sends, to the second electronic device, first screen background image information of a first screen background image obtained by photographing the display screen of the second electronic device using the first location information; The second electronic device determines the first display parameter based on the first screen background image information.

10. The method according to claim 9, characterized in that The first screen background image information includes a plurality of first image feature points of the first screen background image; and The second electronic device determines the first display parameter based on the first screen background image information, including: The second electronic device determines the first display parameter based on the multiple first image feature points of the first screen background image and multiple second image feature points corresponding to the multiple first image feature points in the first screen image of the current display interface of the second electronic device.

11. The method according to claim 10, characterized in that The first display parameter includes the first display ratio, and The second electronic device determines the first display parameter based on the multiple first image feature points of the first screen background image and multiple second image feature points corresponding to the multiple first image feature points in the first screen image of the current display interface of the second electronic device, including: Obtaining a spatial transformation matrix according to the plurality of first image feature points and the plurality of second image feature points; Determining, from a first screen image of a current display interface of the second electronic device according to the spatial transformation matrix, a second center point and a second offset point corresponding to a first center point and a first offset point of the first screen background image; A first distance is calculated according to the coordinates of the first center point and the coordinates of the first offset point, and a second distance is calculated according to the coordinates of the second center point and the coordinates of the second offset point; The first display ratio is obtained according to a ratio between the first distance and the second distance.

12. The method according to claim 10, characterized in that The first display parameter includes the first display angle, and The second electronic device determines the first display parameter based on the multiple first image feature points of the first screen background image and multiple second image feature points corresponding to the multiple first image feature points in the first screen image of the current display interface of the second electronic device, including: Obtaining a spatial transformation matrix according to the plurality of first image feature points and the plurality of second image feature points; Determining, from a first screen image of a current display interface of the second electronic device according to the spatial transformation matrix, a second center point and a second offset point corresponding to a first center point and a first offset point of the first screen background image; Calculating a first angle between a straight line where the first center point and the first offset point are located and a first coordinate axis in a coordinate system of a second electronic device, and calculating a second angle between a straight line where the second center point and the second offset point are located and the first coordinate axis in the coordinate system of the second electronic device; The first display angle is obtained according to the difference between the first angle and the second angle.

13. An electronic device, characterized in that: include: one or more processors; One or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device executes the display method described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The storage medium stores instructions, which, when executed on a computer, cause the computer to execute the display method according to any one of claims 1 to 12.

15. A computer program product comprising instructions, characterized in that When the computer program product is executed on a computer, the computer is enabled to execute the display method according to any one of claims 1 to 12.

Images