Image processing method, image processing device, electronic equipment and medium

Abstract

The invention discloses an image processing method, an image processing device, electronic equipment and a medium, and belongs to the technical field of image processing. The image processing method comprises the following steps: receiving a first parameter sent by second electronic equipment, wherein the first parameter comprises a first pose parameter of the second electronic equipment; based on the first pose parameter and a pose conversion parameter, obtaining a first picture, the pose conversion parameter being used for converting a pose parameter of the second electronic device; and sending a first message to the second electronic equipment, wherein the first message comprises the first picture.

Description

Technical Field

[0001] This application belongs to the technical field of image processing, and particularly relates to an image processing method, an image processing device, an electronic device, and a medium. Background Art

[0002] Extended Reality (XR) is a comprehensive technology that encompasses Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). XR technology creates an immersive user experience by integrating computer-generated content with the real world.

[0003] Currently, if an XR device needs to share the virtual screen it displays in real time to other mobile devices, the XR device can first establish a connection with the mobile device via Wi-Fi or Bluetooth, and then the XR device shares the virtual screen it displays with the mobile device through the established connection, so that the user of the mobile device can view the virtual screen displayed on the XR device through the mobile device.

[0004] However, through the above method, since the virtual screen viewed by the user through the mobile device is rendered from the viewing angle of the user of the XR device, the virtual screen displayed on the mobile device cannot meet the immersive experience of the user, that is, the immersive effect of the virtual screen displayed on the mobile device is poor. Summary of the Invention

[0005] The purpose of the embodiments of this application is to provide an image processing method, an image processing device, an electronic device, and a medium, which can improve the immersive effect of the virtual screen displayed on the electronic device.

[0006] In a first aspect, the embodiments of this application provide an image processing method, which is applied to a first electronic device. The method includes:

[0007] Receiving a first parameter sent by a second electronic device, where the first parameter includes a first pose parameter of the second electronic device;

[0008] Obtaining a first screen based on the first pose parameter and a pose conversion parameter, where the pose conversion parameter is used to convert the pose parameter of the second electronic device;

[0009] Sending a first message to the second electronic device, where the first message includes the first screen.

[0010] In a second aspect, the embodiments of this application provide an image processing method, which is applied to a second electronic device. The method includes:

[0011] Send the first parameter of the second electronic device to the first electronic device, where the first parameter includes the first pose parameter of the second electronic device, and the first pose parameter is used for the first electronic device to obtain a picture;

[0012] Receive a first message sent by the first electronic device, where the first message includes a first picture;

[0013] After receiving multiple first messages, generate a second picture based on the multiple first pictures.

[0014] In a third aspect, an embodiment of the present application provides an image processing apparatus, and the apparatus includes:

[0015] A receiving module, configured to receive a first parameter sent by a second electronic device, where the first parameter includes a first pose parameter of the second electronic device;

[0016] An obtaining module, configured to obtain a first picture based on the first pose parameter and a pose conversion parameter, where the pose conversion parameter is used to convert the pose parameter of the second electronic device;

[0017] A sending module, configured to send a first message to the second electronic device, where the first message includes the first picture.

[0018] In a fourth aspect, an embodiment of the present application provides an image processing apparatus, and the apparatus includes:

[0019] A sending module, configured to send the first parameter of the second electronic device to the first electronic device, where the first parameter includes the first pose parameter of the second electronic device, and the first pose parameter is used for the first electronic device to obtain a picture;

[0020] A receiving module, configured to receive a first message sent by the first electronic device, where the first message includes the first picture;

[0021] A processing module, configured to generate a second picture based on the multiple first pictures after receiving multiple first messages.

[0022] In a fifth aspect, an embodiment of the present application provides an electronic device, and the electronic device includes a processor and a memory. The memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method described in the first aspect are implemented.

[0023] In a sixth aspect, an embodiment of the present application provides an electronic device, and the electronic device includes a processor and a memory. The memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method described in the second aspect are implemented.

[0024] In a seventh aspect, an embodiment of the present application provides a readable storage medium, on which a program or instructions are stored, and when the program or instructions are executed by a processor, the steps of the image processing method described in the first aspect are implemented, or the steps of the image processing method described in the second aspect are implemented.

[0025] In an eighth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run a program or instructions to implement the image processing method described in the first aspect, or to implement the image processing method described in the second aspect.

[0026] In a ninth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the image processing method described in the first aspect, or to implement the image processing method described in the first aspect.

[0027] In the embodiment of the present application, the first electronic device receives a first parameter sent by the second electronic device. The first parameter includes the first pose parameter of the second electronic device; based on the first pose parameter and the pose conversion parameter, a first image is obtained, and the pose conversion parameter is used to convert the pose parameter of the second electronic device; a first message is sent to the second electronic device, and the first message includes the first image. In this way, the first electronic device can obtain an image based on the pose parameter of the second electronic device and the pose conversion parameter, so as to obtain an image that matches the viewing angle of the user of the second electronic device. When the first electronic device sends the image to the second electronic device, the second electronic device can display an image that matches the user's viewing angle, and further, the image displayed by the electronic device can meet the user's immersive experience, improving the immersive effect of the image displayed by the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0029] Figure 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0030] Figure 3 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0031] Figure 4 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0032] Figure 5 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0033] Figure 6 It is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0034] Figure 7 It is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0035] Figure 8 It is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0036] Figure 9 It is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0037] Figure 10 It is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0038] Figure 11A It is a schematic diagram of an image processing process provided by an embodiment of the present application;

[0039] Figure 11B It is a schematic diagram of an image processing process provided by an embodiment of the present application;

[0040] Figure 11C It is a schematic diagram of an image processing process provided by an embodiment of the present application;

[0041] Figure 11D It is a schematic diagram of an image processing process provided by an embodiment of the present application;

[0042] Figure 11E It is a schematic diagram of an image processing process provided by an embodiment of the present application;

[0043] Figure 11F It is a schematic diagram of an image processing process provided by an embodiment of the present application;

[0044] Figure 12 It is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application;

[0045] Figure 13 It is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application;

[0046] Figure 14 It is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application;

[0047] Figure 15 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0048] Figure 16 It is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0049] The technical solutions in the embodiments of the present application will be clearly described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application belong to the scope of protection of the present application.

[0050] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order different from those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same category, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally means an "or" relationship between the associated objects before and after.

[0051] The terms "at least one (item)", "at least one of", etc. in the specification and claims of the present application refer to any one, any two or more combinations of the objects they contain. For example, at least one (item) of a, b, and c can represent: "a", "b", "c", "a and b", "a and c", "b and c", and "a, b, and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" means two or more, and its expressed meaning is similar to that of "at least one (item)".

[0052] The image processing method, image processing device, electronic device and medium provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings, through specific embodiments and their application scenarios.

[0053] In the image processing method provided in the embodiments of the present application, the first electronic device receives a first parameter sent by the second electronic device, and the first parameter includes the first pose parameter of the second electronic device; based on the first pose parameter and the pose conversion parameter, obtains a first picture; and sends a first message to the second electronic device, where the first message includes the first picture. In this way, the first electronic device can obtain a picture based on the pose parameter and the pose conversion parameter of the second electronic device, so as to obtain a picture that matches the viewing angle of the user of the second electronic device. When the first electronic device sends the picture to the second electronic device, the second electronic device can display a picture that matches the user's viewing angle, and further, the picture displayed by the electronic device can meet the immersive experience of the user, improving the immersive effect of the picture displayed by the electronic device.

[0054] The execution subject of the image processing method provided by the embodiments of the present application can be an image processing device. Exemplarily, the image processing device can be an electronic device, or a functional component or functional entity in the electronic device. Hereinafter, taking the execution subject as an electronic device as an example, the image processing method provided by the embodiments of the present application will be described by way of example.

[0055] Figure 1 is a schematic flowchart of the image processing method provided by the embodiments of the present application. This image processing method can be applied to a first electronic device, such as Figure 1 As shown, the image processing method provided by the embodiments of the present application may include the following steps 101 to 103.

[0056] Step 101, the first electronic device receives a first parameter sent by the second electronic device.

[0057] In some embodiments of the present application, the above-mentioned first parameter includes the first pose parameter of the above-mentioned second electronic device.

[0058] In some embodiments of the present application, the above-mentioned first pose parameter may be the first pose matrix of the above-mentioned second electronic device in the first coordinate system, and the first coordinate system is the coordinate system where the above-mentioned second electronic device is located.

[0059] In some embodiments of the present application, the above-mentioned first electronic device may be an XR device, such as an XR headset.

[0060] In some embodiments of the present application, the above-mentioned second electronic device may be a device for displaying the virtual screen of the XR device, such as a mobile device for displaying the virtual screen of the XR device.

[0061] In some embodiments of the present application, the first electronic device may receive the first parameter sent by the above-mentioned second electronic device in real time. For example, the first device may receive the first parameter sent by the above-mentioned second electronic device every 5 ms.

[0062] Step 102, the first electronic device obtains a first image based on the first pose parameter and the pose conversion parameter.

[0063] In some embodiments of the present application, the above-mentioned pose conversion parameter is used to convert the first pose parameter of the above-mentioned second electronic device.

[0064] In some embodiments of the present application, the above-mentioned pose conversion parameter may be a pose conversion matrix.

[0065] In some embodiments of the present application, the above-mentioned pose conversion matrix is used to convert the first pose matrix of the above-mentioned second electronic device from the first coordinate system to the second coordinate system, and the second coordinate system is the coordinate system where the above-mentioned first electronic device is located.

[0066] In some embodiments of the present application, the above-mentioned first screen may be a virtual screen in the virtual space of the above-mentioned first electronic device.

[0067] In some embodiments of the present application, the above-mentioned virtual space may be a virtual space under the above-mentioned second coordinate system.

[0068] In some embodiments of the present application, in combination Figure 1 , such as Figure 2 shown, the above-mentioned step 102 may be implemented through the following step 102a to the following step 102c:

[0069] Step 102a: The first electronic device obtains the shooting parameters of the virtual camera of the first electronic device based on the first pose parameter and the pose conversion parameter.

[0070] In some embodiments of the present application, the above-mentioned virtual camera is used to shoot the virtual screen of the first electronic device.

[0071] In some embodiments of the present application, the above-mentioned first parameter further includes the first camera projection parameter of the above-mentioned second electronic device, and the shooting parameters include the shooting position of the above-mentioned virtual camera, the shooting angle of the above-mentioned virtual camera, and the second camera projection parameter of the above-mentioned virtual camera; in combination Figure 2 , such as Figure 3 shown, the above-mentioned step 102a may be implemented through the following step 102a1 to the following step 102a3:

[0072] Step 102a1: The first electronic device obtains the second pose parameter based on the first pose parameter and the pose conversion parameter.

[0073] In some embodiments of the present application, the electronic device may use the following formula to obtain the above-mentioned second pose parameter:

[0074] Pose1 = M * Pose0; (1)

[0075] Wherein, Pose1 represents the above-mentioned second pose parameter, Pose0 represents the above-mentioned first pose parameter, and M represents the above-mentioned pose conversion parameter.

[0076] In some embodiments of the present application, the above-mentioned second pose parameter may be the second pose matrix of the above-mentioned second electronic device in the above-mentioned second coordinate system.

[0077] Step 102a2: The first electronic device determines the shooting position and the shooting angle based on the second pose parameter.

[0078] In some embodiments of the present application, when the first electronic device is an XR device and the first electronic device displays a virtual screen, a virtual space can be constructed in the second coordinate system described above. When the user wearing the XR device moves, the user can view the virtual screen in the virtual space from different angles and positions.

[0079] In some embodiments of the present application, the first electronic device can construct the virtual camera described above and place the virtual camera at Pose1 in the virtual space, that is, the second pose parameter is the shooting position and shooting angle of the virtual camera.

[0080] Step 102a3: The first electronic device determines the first camera projection parameter as the second camera projection parameter.

[0081] In some embodiments of the present application, the first camera projection parameter can be the camera projection matrix of the second electronic device, and the camera projection matrix can be the camera projection matrix of the camera of the second electronic device in the first coordinate system.

[0082] In some embodiments of the present application, the second camera projection parameter can be the camera projection matrix of the first electronic device, and the camera projection matrix can be the camera projection matrix of the camera of the first electronic device in the second coordinate system.

[0083] In some embodiments of the present application, the first electronic device can use the camera projection matrix as the camera projection matrix of the virtual camera in the virtual space.

[0084] In this way, the first electronic device obtains the second pose parameter based on the first pose parameter and the pose conversion parameter, determines the shooting position and shooting angle based on the second pose parameter, and determines the first camera projection parameter as the second camera projection parameter, so that the first electronic device can render a picture that matches the viewing angle of the user of the second electronic device based on the shooting position, shooting angle, and the second camera projection parameter. Therefore, when the picture rendered from the captured image is sent to the second electronic device, the second electronic device can display a picture that matches the user's viewing angle, thereby enabling the picture displayed by the electronic device to meet the user's immersive experience and improving the immersive effect of the picture displayed by the electronic device.

[0085] Step 102b: The first electronic device obtains the first spatial element in the virtual space of the first electronic device based on the shooting parameters.

[0086] In some embodiments of the present application, after the first electronic device determines the shooting position, shooting angle, and camera projection matrix of the virtual camera in the virtual space, it may obtain a first spatial element corresponding to the shooting position, shooting angle, and camera projection matrix from the virtual space based on the shooting position, shooting angle, and camera projection matrix.

[0087] Step 102c: The first electronic device renders the first spatial element to obtain a first picture.

[0088] In some embodiments of the present application, the first electronic device may render the first spatial element onto an image to obtain the first picture.

[0089] In some embodiments of the present application, the first parameter further includes the screen resolution of the second electronic device; combined Figure 3 , as Figure 4 shown, the above step 102c may be implemented through the following step 102c1:

[0090] Step 102c1: The first electronic device renders the first spatial element based on the screen resolution to obtain a first picture.

[0091] In some embodiments of the present application, the resolution of the first picture is less than the screen resolution.

[0092] In some embodiments of the present application, the electronic device may render the first spatial element at a target resolution to obtain the first picture, and the target resolution is less than the screen resolution.

[0093] It should be noted that when the electronic device renders the first spatial element at a target resolution, the resolution of the obtained first picture is the target resolution.

[0094] For example, when the screen resolution is (ResolutionX, ResolutionY), the target resolution may be (ResolutionX / 2, ResolutionY / 2). The first electronic device may render the first spatial element at (ResolutionX / 2, ResolutionY / 2) to obtain the first picture with a resolution of (ResolutionX / 2, ResolutionY / 2).

[0095] In this way, by rendering the first spatial element based on the screen resolution to obtain a picture with a resolution less than the screen resolution, the first electronic device can reduce the amount of data processing when rendering an image, thereby improving the image rendering efficiency.

[0096] In this way, the first electronic device obtains the shooting parameters of the virtual camera of the first electronic device based on the first pose parameter and the pose conversion parameter; obtains the first spatial element in the virtual space of the first electronic device based on the shooting parameters; renders the first spatial element to obtain the first picture, so that the first electronic device can render a picture that matches the viewing angle of the user of the second electronic device. Therefore, when the rendered picture is sent to the second electronic device, the second electronic device can display a picture that matches the viewing angle of the user, and further enables the picture displayed by the electronic device to meet the immersive experience of the user, improving the immersive effect of the picture displayed by the electronic device.

[0097] Step 103: The first electronic device sends a first message to the second electronic device.

[0098] In some embodiments of the present application, the above first message includes the above first picture.

[0099] In some embodiments of the present application, the above first message further includes the above second pose parameter.

[0100] In some embodiments of the present application, the first electronic device can send the first message to the above second electronic device in real time.

[0101] In some embodiments of the present application, every time the first electronic device receives the first parameter of the above second electronic device, it can execute the above step 102 and the above step 103, and after the above step 102 and the above step 103, send a first message to the above second electronic device. Each first message includes the first picture and the second pose parameter.

[0102] In the image processing method provided in the embodiments of the present application, the first electronic device receives the first parameter sent by the second electronic device. The first parameter includes the first pose parameter of the second electronic device; obtains the first picture based on the first pose parameter and the pose conversion parameter; sends a first message to the second electronic device. The first message includes the first picture. In this way, the first electronic device can obtain a picture based on the pose parameter and the pose conversion parameter of the second electronic device, so as to obtain a picture that matches the viewing angle of the user of the second electronic device. When the first electronic device sends the picture to the second electronic device, the second electronic device can display a picture that matches the viewing angle of the user, and further the picture displayed by the electronic device can meet the immersive experience of the user, improving the immersive effect of the picture displayed by the electronic device.

[0103] In some embodiments of the present application, before the above step 101, the image processing method provided in the embodiments of the present application may further include the following step 104:

[0104] Step 104: The first electronic device identifies and locates the first marker image displayed by the second electronic device to obtain the pose conversion parameters.

[0105] In some embodiments of the present application, the above first marker image is used to indicate the position of the above second electronic device.

[0106] In some embodiments of the present application, the above first marker image can be an Apriltag picture, and the first electronic device can identify and locate the Apriltag picture displayed by the second electronic device through the Apriltag positioning algorithm to obtain the above pose conversion parameters.

[0107] In this way, the first electronic device identifies and locates the first marker image displayed by the second electronic device to obtain the pose conversion parameters, and can quickly and accurately obtain the pose conversion parameters.

[0108] In some embodiments of the present application, before the above step 101, the image processing method provided by the embodiments of the present application may further include the following steps 105a to the following step 105c:

[0109] Step 105a: The first electronic device identifies and locates the second marker image at a preset position to obtain the third pose parameter.

[0110] In some embodiments of the present application, the above second marker image can be a two-dimensional image with the number of feature points exceeding the first threshold.

[0111] In some embodiments of the present application, the first electronic device can use a two-dimensional image recognition and tracking algorithm to identify and locate the above two-dimensional image to obtain the above third pose parameter.

[0112] In some embodiments of the present application, the above third pose parameter can be a pose matrix of the first electronic device relative to the above two-dimensional image.

[0113] Step 105b: The first electronic device obtains the fourth pose parameter.

[0114] In some embodiments of the present application, the above fourth pose parameter is obtained by the second electronic device identifying and locating the above second marker image.

[0115] In some embodiments of the present application, in the case where the above second marker image is a two-dimensional image with the number of feature points exceeding the first threshold, the above fourth pose parameter can be obtained by the second electronic device using a two-dimensional image recognition and tracking algorithm to identify and locate the above two-dimensional image.

[0116] In some embodiments of the present application, the above fourth pose parameter can be a pose matrix of the second electronic device relative to the above two-dimensional image.

[0117] Step 105c: The first electronic device obtains a pose conversion parameter based on the third pose parameter and the fourth pose parameter.

[0118] In some embodiments of the present application, the electronic device may use the following formula to obtain the above-mentioned pose conversion parameter:

[0119] M = M1 -1 *M0; (2)

[0120] where M1 represents the above-mentioned third pose parameter, and M0 represents the above-mentioned fourth pose parameter.

[0121] In this way, the first electronic device can quickly and accurately obtain the pose conversion parameter by identifying and positioning the second marker image at the preset position, obtaining the fourth pose parameter, and obtaining the pose conversion parameter based on the third pose parameter and the fourth pose parameter.

[0122] In some embodiments of the present application, before the above-mentioned step 101, the image processing method provided by the embodiments of the present application may further include the following step 106:

[0123] Step 106: The first electronic device sends the screen background data to the second electronic device.

[0124] In some embodiments of the present application, the above-mentioned screen background data may be the screen background data corresponding to the screen background of the entire virtual space.

[0125] In some embodiments of the present application, the format of the above-mentioned screen background data may be common 3D model file data such as FBX format, OBJ format, Graphics Language Transmission Format (GLTF), Graphics Language Transmission Format Binary (GLB), Universal Scene Description (USD), Universal Scene Description Package (USDZ), Universal Scene Description Asset (USDA), etc.

[0126] It should be noted that the FBX format is a proprietary file format developed by Kaydara and owned by Autodesk since 2006. It is a very popular 3D model format widely used in industries such as manufacturing, engineering construction, and media and entertainment to assist users in these industries with 3D visualization design. The OBJ format is a standard 3D model file format developed by Alias Wavefront for its set of workstation-based 3D modeling and animation software. The OBJ file format has a simple structure and supports almost all 3D modeling software.

[0127] In some embodiments of the present application, the above-mentioned screen background is used for the second electronic device to generate a screen background.

[0128] In this way, by sending the screen background data from the first electronic device to the second electronic device, on the one hand, it enables the second electronic device to generate a screen background; on the other hand, it avoids excessive data transmission caused by directly transmitting the screen background rendered from the screen background data, effectively reducing the data transmission volume and improving the transmission efficiency.

[0129] Figure 5 It is a schematic flowchart of the image processing method provided by the embodiments of the present application. This image processing method can be applied to the second electronic device, such as Figure 5 As shown, the image processing method provided by the embodiments of the present application may include the following steps 201 to step 203.

[0130] Step 201: The second electronic device sends the first parameter of the second electronic device to the first electronic device.

[0131] In some embodiments of the present application, the above-mentioned first parameter includes the first pose parameter of the second electronic device, and the first pose parameter is used for the first electronic device to acquire an image.

[0132] In some embodiments of the present application, the above-mentioned first parameter may further include the screen resolution of the second electronic device and the camera projection parameter of the second electronic device.

[0133] Step 202: The second electronic device receives the first message sent by the first electronic device.

[0134] In some embodiments of the present application, the above-mentioned first message includes the first image.

[0135] In some embodiments of the present application, the second electronic device may receive the first message sent by the first electronic device in real time. For example, the second device may receive the first message sent by the second electronic device every 5 ms.

[0136] Step 203. After receiving multiple first messages, the second electronic device generates a second screen based on the multiple first screens.

[0137] In some embodiments of the present application, each first message further includes a second pose parameter; combined Figure 5 , such as Figure 6 shown, the above step 203 can be implemented through the following step 203a:

[0138] Step 203a. The second electronic device generates a second screen based on the multiple first screens and the multiple second pose parameters.

[0139] In some embodiments of the present application, combined Figure 6 , such as Figure 7 shown, the above step 203a can be implemented through the following step 203a1 to the following step 203a3:

[0140] Step 203a1. The second electronic device determines, based on the second pose parameter corresponding to the first target screen and the second pose parameter corresponding to the second target screen, a first pixel point corresponding to each pixel point of the first target screen in the second target screen.

[0141] In some embodiments of the present application, the above first target screen is any one of the multiple first screens, and the above second target screen is a screen other than the first target screen among the multiple first screens.

[0142] In some embodiments of the present application, the second pose parameter corresponding to the first target screen may represent the shooting position when the virtual camera acquires the first target screen, and the second pose parameter corresponding to the second target screen may represent the shooting position when the virtual camera acquires the second target screen.

[0143] In some embodiments of the present application, the electronic device may obtain a first pose deviation between the second pose parameter corresponding to the first target screen and the second pose parameter corresponding to the second target screen, and then determine, in the second target screen, a pixel point whose pose deviation from each pixel point of the first target screen is the first pose deviation. The pixel point in the second target screen whose pose deviation from each pixel point of the first target screen is the first pose deviation is the first pixel point corresponding to each pixel point of the first target screen in the second target screen.

[0144] Preferably, the above first target screen may be the last first screen among the multiple first screens.

[0145] Step 203a2. The second electronic device performs weighted averaging on the first pixel points in the second target screen and the corresponding pixel points in the first target screen to obtain a third screen.

[0146] In some embodiments of the present application, the second electronic device performs weighted averaging on the pixel value of the first pixel point in the second target screen and the pixel value of the corresponding pixel point in the first target screen to obtain the pixel value of each pixel point in the third screen.

[0147] Step 203a3: The second electronic device performs interpolation operation on the third screen based on the screen resolution of the second electronic device to obtain the second screen.

[0148] In some embodiments of the present application, the resolution of the above second screen is the above screen resolution.

[0149] In some embodiments of the present application, the second electronic device may use a super-resolution algorithm to perform interpolation operation on the above third screen to obtain a second screen with a resolution of the above screen resolution.

[0150] For example, when the above screen resolution can be (ResolutionX, ResolutionY), the resolution of the interpolation operation of the above third screen can be (ResolutionX / 2, ResolutionY / 2). The second electronic device may use a super-resolution algorithm to perform double interpolation operation on the third screen with a resolution of (ResolutionX / 2, ResolutionY / 2) to obtain a second screen with a resolution of (ResolutionX, ResolutionY).

[0151] In this way, the second electronic device determines, based on the second pose parameter corresponding to the first target screen and the second pose parameter corresponding to the second target screen, the first pixel point corresponding to each pixel point of the first target screen in the second target screen; performs weighted averaging on the first pixel point in the second target screen and the corresponding pixel point in the first target screen to obtain the third screen; and performs interpolation operation on the third screen based on the screen resolution of the second electronic device to obtain the second screen, which can improve the resolution of the screen and thus improve the display effect of the screen.

[0152] In the image processing method provided by the embodiments of the present application, the second electronic device sends the first parameter of the second electronic device to the first electronic device, where the first parameter includes the first pose parameter of the second electronic device, and the first pose parameter is used for the first electronic device to obtain a picture; receives the first message sent by the first electronic device, where the first message includes the first picture; and after receiving a plurality of first messages, generates a second picture based on the plurality of first pictures. In this way, the second electronic device can generate a second picture based on the plurality of pictures obtained by the first electronic device according to the pose parameter of the second electronic device, so that the second electronic device can generate a picture that matches the viewing angle of the user of the second electronic device, and further enables the picture displayed by the electronic device to meet the immersive experience of the user, improving the immersive effect of the picture displayed by the electronic device.

[0153] In some embodiments of the present application, in combination with Figure 5 , as Figure 8 shown, before the above step 201, the image processing method provided by the embodiments of the present application may further include the following step 204, and after the above step 203, the method provided by the embodiments of the present application may further include the following step 205:

[0154] Step 204: The second electronic device receives the picture background data sent by the first electronic device.

[0155] It should be noted that the specific explanation of the above picture background data can refer to the relevant description in the above step 107, and will not be elaborated here in this embodiment.

[0156] Step 205: The second electronic device generates and displays a fourth picture based on the second picture and the picture background data.

[0157] In some embodiments of the present application, the above picture background data includes virtual picture background data; the above step 205 can be implemented through the following step 205a and the following step 205b:

[0158] Step 205a: The second electronic device renders the virtual picture background data based on the second target pose parameter to obtain a virtual picture background.

[0159] In some embodiments of the present application, the above second target pose parameter is any one of the second pose parameters carried by the plurality of first messages.

[0160] Preferably, the above target parameter may be the second pose parameter carried by the last first message among the above plurality of first messages.

[0161] Step 205b: The second electronic device fuses the second picture and the virtual picture background to obtain and display a fourth picture.

[0162] In this way, the second electronic device renders the virtual screen background data according to the second target pose parameter to obtain the virtual screen background, and fuses the second screen and the virtual screen background to obtain and display the fourth screen, so that the second electronic device can generate a virtual screen background that matches the user's viewing angle. Furthermore, the second screen and the virtual screen background are fused to obtain and display the fourth screen, which can display a screen that matches the user's viewing angle, and thus the screen displayed by the electronic device can meet the user's immersive experience, improving the immersive effect of the screen displayed by the electronic device.

[0163] In some embodiments of the present application, the above screen background data further includes an occupancy ratio, which is used to represent the proportion of the above virtual screen background in the screen background. The above step 205b can be implemented by the following step 205b1 or the following step 205b2:

[0164] Step 205b1: When the occupancy ratio is greater than or equal to the first threshold, the second electronic device fuses the second screen and the virtual screen background to obtain and display the fourth screen.

[0165] In some embodiments of the present application, when the occupancy ratio is greater than or equal to the first threshold, it indicates that the proportion of the virtual screen background in the screen background is large, and the proportion of the real screen background in the screen background is small and can be ignored. Therefore, the second electronic device does not need to obtain the real screen background, and fusing the above second screen and the above virtual screen background can obtain the screen to be displayed, that is, the above fourth screen.

[0166] Step 205b2: When the occupancy ratio is greater than the second threshold and less than the first threshold, the second electronic device captures a second image through the camera, and fuses the second image and the virtual screen background to obtain a fused screen background, and fuses the fused screen background and the second screen to obtain and display the fourth screen.

[0167] In some embodiments of the present application, the above second threshold is less than the above first threshold.

[0168] In some embodiments of the present application, when the occupancy ratio is greater than the second threshold and less than the first threshold, it indicates that the proportion of the virtual screen background and the real screen background in the screen background is small. Therefore, the second electronic device can capture a second image through the camera, fuse the above second image as the real screen background and the above virtual screen background to obtain a fused screen background, and then fuse the above fused screen background and the above second screen to obtain the screen to be displayed, that is, the above fourth screen.

[0169] In this way, when the occupation ratio is greater than or equal to the first threshold, the second electronic device fuses the second screen and the virtual screen background to obtain and display the fourth screen, which can generate a screen including the screen content and the virtual screen background, thereby improving the display effect of the electronic device on the screen; when the occupation ratio is greater than the second threshold and less than the first threshold, the second electronic device captures a second image through the camera, fuses the second image and the virtual screen background to obtain a fused screen background, and fuses the fused screen background and the second screen to obtain and display the fourth screen, which can generate a screen including the screen content, the virtual screen background and the real background captured by the camera, thereby improving the display effect of the electronic device on the screen.

[0170] Next, in combination with Figure 9 , the image processing method provided in the embodiments of the present application will be further described. As Figure 9 shown, the image processing method provided in the embodiments of the present application may include the following steps:

[0171] Step 301, the first electronic device sends screen background data to the second electronic device.

[0172] In some embodiments of the present application, the above screen background data includes virtual screen background data and an occupation ratio, and the occupation ratio is used to represent the proportion of the virtual screen background in the screen background.

[0173] It should be noted that the specific implementation process of the above step 301 can refer to the above step 106. To avoid repetition, it will not be elaborated here in this embodiment.

[0174] Step 302, the second electronic device receives the screen background data sent by the first electronic device.

[0175] It should be noted that the specific implementation process of the above step 302 can refer to the above step 204. To avoid repetition, it will not be elaborated here in this embodiment.

[0176] Step 303, the second electronic device sends the first parameter of the second electronic device to the first electronic device.

[0177] In some embodiments of the present application, the above first parameter includes the first pose parameter of the second electronic device, the camera projection parameter of the second electronic device, and the screen resolution, and the first parameter is used for the first electronic device to obtain the screen.

[0178] It should be noted that the specific implementation process of the above step 303 can refer to the above step 201. To avoid repetition, it will not be elaborated here in this embodiment.

[0179] Step 304, the first electronic device receives the first parameter sent by the second electronic device.

[0180] It should be noted that the specific implementation process of step 304 above can refer to step 101 above. To avoid repetition, it will not be elaborated here in this embodiment.

[0181] Step 305: The first electronic device obtains a second pose parameter based on the first pose parameter and the pose conversion parameter.

[0182] It should be noted that the specific implementation process of step 305 above can refer to step 102a1 above. To avoid repetition, it will not be elaborated here in this embodiment.

[0183] Step 306: The first electronic device determines a shooting position and a shooting angle based on the second pose parameter.

[0184] It should be noted that the specific implementation process of step 306 above can refer to step 102a2 above. To avoid repetition, it will not be elaborated here in this embodiment.

[0185] Step 307: The first electronic device determines the first camera projection parameter as the second camera projection parameter.

[0186] It should be noted that the specific implementation process of step 307 above can refer to step 102a3 above. To avoid repetition, it will not be elaborated here in this embodiment.

[0187] Step 308: The first electronic device obtains a first spatial element in the virtual space of the first electronic device based on the shooting position, the shooting angle, and the second camera projection parameter.

[0188] It should be noted that the specific implementation process of step 308 above can refer to step 102b above. To avoid repetition, it will not be elaborated here in this embodiment.

[0189] Step 309: The first electronic device renders the first spatial element based on the screen resolution to obtain a first picture.

[0190] In some embodiments of the present application, the resolution of the first picture is less than the screen resolution.

[0191] It should be noted that the specific implementation process of step 309 above can refer to step 102c1 above. To avoid repetition, it will not be elaborated here in this embodiment.

[0192] Step 310: The first electronic device sends a first message to the second electronic device.

[0193] In some embodiments of the present application, the first message includes the first picture and the second pose parameter.

[0194] It should be noted that the specific implementation process of step 301 above can refer to step 103 above. To avoid repetition, it will not be elaborated here in this embodiment.

[0195] Step 311: The second electronic device receives the first message sent by the first electronic device.

[0196] It should be noted that the specific implementation process of step 311 above can refer to step 202 above. To avoid repetition, it will not be elaborated here in this embodiment.

[0197] Step 312: After the second electronic device receives multiple first messages, it generates a second picture based on the multiple first pictures and multiple second pose parameters.

[0198] It should be noted that the specific implementation process of step 312 above can refer to step 203a above. To avoid repetition, it will not be elaborated here in this embodiment.

[0199] Step 313: When the occupation ratio is greater than or equal to the first threshold, the second electronic device fuses the second picture and the virtual picture background to obtain and display the fourth picture.

[0200] It should be noted that the specific implementation process of step 313 above can refer to step 205a above. To avoid repetition, it will not be elaborated here in this embodiment.

[0201] Step 314: When the occupation ratio is greater than the second threshold and less than the first threshold, the second electronic device captures a second image through the camera, fuses the second image and the virtual picture background to obtain a fused picture background, and fuses the fused picture background and the second picture to obtain and display the fourth picture.

[0202] In some embodiments of the present application, the second threshold is less than the first threshold.

[0203] It should be noted that the specific implementation process of step 314 above can refer to step 205b above. To avoid repetition, it will not be elaborated here in this embodiment.

[0204] In the image processing method provided in the embodiments of the present application, the first electronic device can render the image captured based on the pose parameters and pose conversion parameters of the second electronic device to obtain a picture that matches the viewing angle of the user of the second electronic device. Thus, when the first electronic device sends the picture to the second electronic device, the second electronic device can display a picture that matches the viewing angle of the user, thereby enabling the picture displayed by the electronic device to meet the immersive experience of the user and improving the immersive effect of the picture displayed by the electronic device.

[0205] Figure 10It is a schematic flowchart of an image processing method provided by an embodiment of this application. Taking the first electronic device as an XR device and the second electronic device as a mobile device as an example, the method may include the following steps 401 to step 405 below.

[0206] Step 401: The XR device obtains the real-time pose matrix of the mobile device (i.e., the above-mentioned first pose parameter).

[0207] Exemplarily, as Figure 11A shown, the mobile device can run the client program on the mobile device, search for and connect to the XR device through the local area network. After connecting to the XR device, the mobile device can enable the Visual Simultaneous Localization and Mapping (ViSLAM) function. After the ViSLAM is successfully enabled, the XR device can obtain the real-time pose matrix Pose0 of the mobile device, and the pose matrix Pose0 is determined based on the Cartesian coordinate system Coords0 (i.e., the above-mentioned first coordinate system) defined when the mobile device runs.

[0208] Step 402: The XR device obtains the pose transformation matrix (i.e., the above-mentioned pose transformation parameter), and calibrates the pose matrix of the mobile device by using the pose transformation matrix.

[0209] Exemplarily, a Cartesian coordinate system Coords1 (i.e., the above-mentioned second coordinate system) is defined when the XR device runs, and Coords0 and Coords1 are completely independent; therefore, the XR device can use the pose transformation matrix M to convert the pose matrix Pose0 in the Coords0 coordinate system into the pose matrix Pose1 in the Coords1 coordinate system (i.e., the above-mentioned second pose parameter) to calibrate the pose matrix of the mobile device, that is, Pose1 = M * Pose0.

[0210] Exemplarily, the XR device can identify and locate a pre-defined marker map by means of image recognition to obtain the pose transformation matrix M. There are two types of marker maps: (1) Apriltag pictures, and the Apriltag positioning algorithm can be used to locate and track the pictures; (2) ordinary two-dimensional pictures, and the two-dimensional image recognition and tracking algorithm can be used to locate and track the pictures.

[0211] Exemplarily, when the marker map is an Apriltag picture, combined with Figure 11B shown, after the mobile device successfully enables ViSLAM, it can display a pre-defined marker map on the screen. The XR device can turn on the camera and aim the camera at the mobile device screen to identify and locate the marker map to obtain the pose transformation matrix M.

[0212] Exemplarily, when the marker image is an Apriltag image, combine Figure 11C As shown, place the pre-printed marker image at a preset position, align the mobile device with the printed marker image at the preset position, and obtain the pose matrix M0 of the mobile device relative to the marker image (i.e., the above-mentioned fourth pose parameter) through image positioning; align the XR device with the printed marker image at the preset position (i.e., the above-mentioned third pose parameter), and obtain the pose matrix M1 of the XR device relative to the marker image; combining M0 and M1 gives M = M1 -1 *M0.

[0213] Step 403: The XR device synchronizes the background data of the virtual space (i.e., the above-mentioned virtual screen background data) to the mobile device.

[0214] Exemplarily, the above background data screen background supports common 3D model file formats such as FBX, OBJ, GLTF / GLB, USD / USDZ / USDA, etc.; in addition, as Figure 11D shown, the XR device can also synchronize the fusion degree data (i.e., the above-mentioned occupancy ratio) of the virtual background (i.e., the above-mentioned model file data) and the real background (i.e., the camera screen) to the mobile device.

[0215] Exemplarily, as Figure 11D shown, the XR device can also acquire and save the key information of the mobile device, including the screen resolution (ResolutionX, ResolutionY), and the camera projection matrix ProjectionMatrix.

[0216] Step 404: The XR device renders the content of the virtual space to obtain a first image.

[0217] Exemplarily, in combination with Figure 11E shown, the XR device can loop execute the following logic at a speed of 30 frames per second:

[0218] Obtain the real-time pose Pose0 of the mobile device, and convert Pose0 into a virtual space pose Pose1 through the pose transformation matrix M;

[0219] Construct a virtual camera, place the virtual camera pose at Pose1 in the virtual space, and set the virtual camera projection matrix to ProjectionMatrix;

[0220] Render the content of the virtual space (i.e., the above-mentioned first space element) onto an image at a resolution of (ResolutionX / 2, ResolutionY / 2).

[0221] It should be noted that the XR device only renders the content of the virtual space and does not render the virtual space background.

[0222] The rendered image (i.e., the second image above) and the pose matrix of the virtual camera during rendering are transmitted to the mobile device through Real-Time Communication (RTC) technology.

[0223] Step 405: The mobile device generates and displays a new image based on the rendered image and the background data.

[0224] Exemplarily, for example Figure 11F As shown, the mobile device continuously receives the latest rendered image frame and pose data, and caches the latest 5 frames of data in the local memory;

[0225] After receiving the latest image frame, in combination with the data of the previous 5 frames, the multi-frame accumulation technology is used to increase the resolution of the real-time image from (ResolutionX / 2, ResolutionY / 2) to (ResolutionX, ResolutionY);

[0226] Exemplarily, the mobile device has saved the virtual space background data synchronized from the XR device, the fusion data of the virtual background and the real background, and the virtual camera pose data; through these data, the mobile device can use computing power to render the background image of the current pose, superimpose the background image and the real-time content image after super-resolution processing, and obtain the final image and display it on the mobile device.

[0227] For the image processing method provided in the embodiments of the present application, on the one hand, the image displayed on the mobile device is the image seen from the perspective of the device where the mobile device is located. This processing enables the mobile device to participate in the virtual space constructed by the XR device, forming a unified and highly interactive interaction experience; on the second hand, since the virtual space background and content are separated, the XR device only renders the virtual space content, and the virtual space content only occupies a part of the image after rendering, and a large part is transparent. Therefore, in the RTC encoding stage, through intra-frame compression technology, the data volume of a frame of image can be effectively reduced; on the third hand, the image of the mobile device is rendered by the XR device, and there is no need to install an application program on the mobile device that is the same as the content of the XR device, so the versatility is strong; on the fourth hand, when the XR device renders the image of the mobile device, the rendered resolution is only 1 / 4 of the mobile device screen, and then through super-resolution processing on the mobile device, the rendering consumption of the XR device can be effectively reduced.

[0228] It should be noted that the above-mentioned various method embodiments, or various possible implementation manners in the various method embodiments, can be executed independently, or any two or more of them can be combined with each other, which can be specifically determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0229] The image processing method provided by the embodiments of the present application may be executed by an image processing device. In the embodiments of the present application, taking the image processing device executing the image processing method as an example, the image processing device provided by the embodiments of the present application is described.

[0230] Figure 12 FIG. 4 is a schematic structural diagram of an image processing device 1200 provided by the embodiments of the present application. The image processing device 1200 includes: a receiving module 1201, an obtaining module 1202, and a sending module 1203.

[0231] Among them, the receiving module 1201 is configured to receive a first parameter sent by a second electronic device, and the first parameter includes a first pose parameter of the second electronic device;

[0232] The obtaining module 1202 is configured to obtain a picture based on the first pose parameter and a pose conversion parameter, where the pose conversion parameter is used to convert the pose parameter of the second electronic device;

[0233] The sending module 1203 is configured to send a first message to the second electronic device, and the first message includes the first picture.

[0234] In some embodiments of the present application, the obtaining module 1202 is specifically configured to:

[0235] Based on the first pose parameter and the pose conversion parameter, obtain shooting parameters of a virtual camera of a first electronic device, where the virtual camera is used to shoot a virtual picture of the first electronic device;

[0236] Based on the shooting parameters, obtain a first spatial element in the virtual space of the first electronic device;

[0237] Render the first spatial element to obtain the first picture.

[0238] In some embodiments of the present application, the first parameter further includes a first camera projection parameter of the second electronic device, and the shooting parameters include a shooting position of the virtual camera, a shooting angle of the virtual camera, and a second camera projection parameter of the virtual camera;

[0239] The obtaining module 1202 is specifically configured to:

[0240] Based on the first pose parameter and the pose conversion parameter, obtain a second pose parameter;

[0241] Based on the second pose parameter, determine the shooting position and the shooting angle;

[0242] Determine the first camera projection parameter as the second camera projection parameter.

[0243] In some embodiments of the present application, the first parameter further includes the screen resolution of the second electronic device;

[0244] The obtaining module 1202 is specifically configured to:

[0245] Render the first spatial element based on the screen resolution to obtain the first picture, and the resolution of the first picture is less than the screen resolution.

[0246] In some embodiments of the present application, in combination Figure 12 , such as Figure 13 shown, the device 1200 further includes:

[0247] A processing module 1204, configured to identify and locate a first marker map displayed on the second electronic device to obtain the pose conversion parameter, where the first marker map is used to indicate the position of the second electronic device.

[0248] In some embodiments of the present application, the processing module 1204 is further configured to identify and locate a second marker map at a preset position to obtain a third pose parameter;

[0249] Obtain a fourth pose parameter, where the fourth pose parameter is obtained by the second electronic device identifying and locating the second marker map;

[0250] Based on the third pose parameter and the fourth pose parameter, obtain the pose conversion parameter.

[0251] In some embodiments of the present application, the sending module 1203 is further configured to:

[0252] Send picture background data to the second electronic device.

[0253] In some embodiments of the present application, the second pose parameter is further included in the first message.

[0254] In the image processing apparatus provided in the embodiments of the present application, by receiving a first parameter sent by a second electronic device, the first parameter includes a first pose parameter of the second electronic device; based on the first pose parameter and a pose conversion parameter, a first image is obtained, and the pose conversion parameter is used to convert the pose parameter of the second electronic device; a first message is sent to the second electronic device, and the first message includes the first image. In this way, the image processing apparatus can obtain an image based on the pose parameter and the pose conversion parameter of the second electronic device, so as to obtain an image that matches the viewing angle of the user of the second electronic device. When the image processing apparatus sends the image to the second electronic device, the second electronic device can display an image that matches the viewing angle of the user, and further, the image displayed by the electronic device can meet the immersive experience of the user, improving the immersive effect of the image displayed by the electronic device.

[0255] The data processing apparatus provided in the embodiments of the present application can implement Figures 1 to 4 each process implemented by the above-mentioned image processing method embodiment and achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0256] Figure 14 Fig. shows a possible structural schematic diagram of the image processing apparatus involved in the embodiments of the present application. As Figure 14 shown, the image processing apparatus 1400 may include:

[0257] A sending module 1401, configured to send a first parameter of a second electronic device to a first electronic device, the first parameter includes a first pose parameter of the second electronic device, and the first pose parameter is used for the first electronic device to obtain an image;

[0258] A receiving module 1402, configured to receive a first message sent by the first electronic device, the first message includes a first image;

[0259] A processing module 1403, configured to generate a second image based on a plurality of first images after receiving a plurality of first messages.

[0260] In some embodiments of the present application, each first message further includes a second pose parameter;

[0261] The processing module 1403 is specifically configured to:

[0262] Generate a second image based on a plurality of first images and a plurality of second pose parameters.

[0263] In some embodiments of the present application, the processing module 1403 is specifically configured to:

[0264] Based on the second pose parameters corresponding to the first target image and the second pose parameters corresponding to the second target image, determine the first pixel points in the second target image corresponding to each pixel point in the first target image, where the first target image is any one of the multiple first images, and the second target image is the image among the multiple first images other than the first target image;

[0265] Perform weighted averaging on the first pixel points in the second target image and the corresponding pixel points in the first target image to obtain a third image;

[0266] Based on the screen resolution of the second electronic device, perform interpolation operation on the third image to obtain the second image.

[0267] In some embodiments of the present application, the receiving module 1402 is further configured to:

[0268] Receive the image background data sent by the first electronic device;

[0269] The processing module is further configured to generate and display a fourth image based on the second image and the image background data.

[0270] In some embodiments of the present application, the image background data includes virtual image background data;

[0271] The processing module 1403 is specifically configured to:

[0272] Render the virtual image background data based on the second target pose parameter to obtain a virtual image background, where the second target pose parameter is any one of the second pose parameters carried by the multiple first messages;

[0273] Fuse the second image and the virtual image background to obtain and display the fourth image.

[0274] In some embodiments of the present application, the image background data further includes an occupancy ratio, and the occupancy ratio is used to represent the proportion of the virtual image background in the image background;

[0275] The processing module 1403 is specifically configured to:

[0276] In the case where the occupancy ratio is greater than or equal to a first threshold, fuse the second image and the virtual image background to obtain and display the fourth image;

[0277] When the occupancy ratio is greater than the second threshold and less than the first threshold, a second image is collected by a camera, and the second image is fused with the virtual screen background to obtain a fused screen background, and the fused screen background is fused with the second screen to obtain and display the fourth screen, where the second threshold is less than the first threshold.

[0278] In the image processing apparatus provided in the embodiments of the present application, by sending a first parameter of a second electronic device to a first electronic device, the first parameter includes a first pose parameter of the second electronic device, and the first pose parameter is used for the first electronic device to obtain a screen; receiving a first message sent by the first electronic device, the first message includes a first screen; after receiving a plurality of first messages, a second screen is generated based on the plurality of first screens. In this way, the second electronic device can generate a second screen based on a plurality of screens obtained by the first electronic device according to the pose parameter of the second electronic device, thereby enabling the second electronic device to generate a screen that matches the viewing angle of the user of the second electronic device, and further enabling the screen displayed by the electronic device to meet the immersive experience of the user, and improving the immersive effect of the screen displayed by the electronic device.

[0279] The image processing apparatus in the embodiments of the present application may be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than the terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a mobile Internet device, an augmented reality / virtual reality device, a robot, a first electronic device, a super mobile personal computer, a netbook, or a personal digital assistant, etc., and may also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0280] The image processing apparatus in the embodiments of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0281] The image processing apparatus provided in the embodiments of the present application can implement each process implemented by each embodiment of the above image processing method. To avoid repetition, it will not be described in detail here.

[0282] Optionally, as Figure 15As shown in the figure, an embodiment of the present application further provides an electronic device 1500, including a processor 1501 and a memory 1502. A program or instruction that can run on the processor 1501 is stored on the memory 1502. When the program or instruction is executed by the processor 1501, each step of the above-described embodiment of the image processing method is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be elaborated here.

[0283] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0284] Figure 16 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application.

[0285] The electronic device 1600 includes, but is not limited to: a radio frequency unit 1601, a network module 1602, an audio output unit 1603, an input unit 1604, a sensor 1605, a display unit 1606, a user input unit 1607, an interface unit 1608, a memory 1609, and a processor 1610, etc.

[0286] Those skilled in the art can understand that the electronic device 1600 may further include a power source (such as a battery) for supplying power to each component. The power source can be logically connected to the processor 1610 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. Figure 16 The structure of the electronic device shown in the figure does not limit the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0287] Among them, the processor 1610 is configured to receive a first parameter sent by a second electronic device, and the first parameter includes a first pose parameter of the second electronic device;

[0288] Based on the first pose parameter and a pose conversion parameter, obtain a first picture, where the pose conversion parameter is used to convert the pose parameter of the second electronic device;

[0289] Send a first message to the second electronic device, where the first message includes the first picture.

[0290] In some embodiments of the present application, the processor 1610 is specifically configured to:

[0291] Based on the first pose parameter and the pose conversion parameter, obtain shooting parameters of a virtual camera of the first electronic device, where the virtual camera is used to obtain a virtual picture of the first electronic device;

[0292] Obtain a first spatial element in the virtual space of the first electronic device based on the shooting parameters;

[0293] Render the first spatial element to obtain the first picture.

[0294] In some embodiments of the present application, the first parameter further includes the first camera projection parameter of the second electronic device, and the shooting parameters include the shooting position of the virtual camera, the shooting angle of the virtual camera, and the second camera projection parameter of the virtual camera;

[0295] The processor 1610 is specifically configured to:

[0296] Obtain a second pose parameter based on the first pose parameter and the pose conversion parameter;

[0297] Determine the shooting position and the shooting angle based on the second pose parameter;

[0298] Determine the first camera projection parameter as the second camera projection parameter.

[0299] In some embodiments of the present application, the first parameter further includes the screen resolution of the second electronic device;

[0300] The processor 1610 is specifically configured to:

[0301] Render the first spatial element based on the screen resolution to obtain the first picture, and the resolution of the first picture is less than the screen resolution.

[0302] In some embodiments of the present application, the processor 1610 is further configured to identify and locate a first marker map displayed on the second electronic device to obtain the pose conversion parameter, and the first marker map is used to indicate the position of the second electronic device.

[0303] In some embodiments of the present application, the processor 1610 is further configured to identify and locate a second marker map at a preset position to obtain a third pose parameter;

[0304] Obtain a fourth pose parameter, where the fourth pose parameter is obtained by the second electronic device identifying and locating the second marker map;

[0305] Obtain the pose conversion parameter based on the third pose parameter and the fourth pose parameter.

[0306] In some embodiments of the present application, the processor 1610 is further configured to send picture background data to the second electronic device.

[0307] In some embodiments of the present application, the second pose parameter is further included in the first message.

[0308] In the electronic device provided in the present application, by receiving a first parameter sent by a second electronic device, the first parameter includes a first pose parameter of the second electronic device; based on the first pose parameter and a pose conversion parameter, a first picture is obtained, and the pose conversion parameter is used to convert the pose parameter of the second electronic device; a first message is sent to the second electronic device, and the first message includes the first picture. In this way, a picture can be obtained based on the pose parameter and the pose conversion parameter of the second electronic device, so as to obtain a picture that matches the viewing angle of the user of the second electronic device. When the first electronic device sends the picture to the second electronic device, the second electronic device can display a picture that matches the viewing angle of the user, thereby enabling the picture displayed by the electronic device to meet the immersive experience of the user and improving the immersive effect of the picture displayed by the electronic device.

[0309] In some embodiments of the present application, the processor 1610 is configured to send a first parameter of the second electronic device to the first electronic device, the first parameter includes a first pose parameter of the second electronic device, and the first pose parameter is used for the first electronic device to obtain a picture;

[0310] Receive a first message sent by the first electronic device, the first message includes a first picture;

[0311] After receiving a plurality of first messages, a second picture is generated based on the plurality of first pictures.

[0312] In some embodiments of the present application, a second pose parameter is further included in each first message;

[0313] The processor 1610 is specifically configured to:

[0314] Generate a second picture based on the plurality of first pictures and the plurality of second pose parameters.

[0315] In some embodiments of the present application, the processor 1610 is specifically configured to:

[0316] Based on the second pose parameter corresponding to the first target picture and the second pose parameter corresponding to the second target picture, determine a first pixel point corresponding to each pixel point of the first target picture in the second target picture, where the first target picture is any picture among the plurality of first pictures, and the second target picture is a picture other than the first target picture among the plurality of first pictures;

[0317] Perform weighted averaging on the first pixel point in the second target picture and the corresponding pixel point in the first target picture to obtain a third picture;

[0318] Interpolate the third picture based on the screen resolution of the second electronic device to obtain the second picture.

[0319] In some embodiments of the present application, the processor 1610 is further configured to receive the picture background data sent by the first electronic device;

[0320] Generate and display a fourth picture based on the second picture and the picture background data.

[0321] In some embodiments of the present application, the picture background data includes virtual picture background data;

[0322] The processor 1610 is specifically configured to:

[0323] Render the virtual picture background data based on the second target pose parameter to obtain a virtual picture background, where the second target pose parameter is any one of the second pose parameters carried by a plurality of first messages;

[0324] Fuse the second picture and the virtual picture background to obtain and display the fourth picture.

[0325] In some embodiments of the present application, the picture background data further includes an occupancy ratio, and the occupancy ratio is used to represent the proportion of the virtual picture background in the picture background;

[0326] The processor 1610 is specifically configured to:

[0327] In the case where the occupancy ratio is greater than or equal to a first threshold, fuse the second picture and the virtual picture background to obtain and display the fourth picture;

[0328] In the case where the occupancy ratio is greater than a second threshold and less than the first threshold, collect a second image through a camera, fuse the second image and the virtual picture background to obtain a fused picture background, and fuse the fused picture background and the second picture to obtain and display the fourth picture, where the second threshold is less than the first threshold.

[0329] In the electronic device provided in the embodiment of the present application, by sending a first parameter of a second electronic device to the first electronic device, the first parameter includes a first pose parameter of the second electronic device, and the first pose parameter is used for the first electronic device to obtain a picture; receiving a first message sent by the first electronic device, the first message includes a first picture; after receiving a plurality of first messages, generating a second picture based on the plurality of first pictures. In this way, the second electronic device can generate a second picture based on the plurality of pictures obtained by the first electronic device according to the pose parameter of the second electronic device, thereby enabling the second electronic device to generate a picture that matches the viewing angle of the user of the second electronic device, and further enabling the picture displayed by the electronic device to meet the immersive experience of the user, and improving the immersive effect of the picture displayed by the electronic device.

[0330] It should be understood that in the embodiment of the present application, the input unit 1604 may include a Graphics Processing Unit (GPU) 16041 and a microphone 16042. The graphics processor 16041 processes the image data of static pictures or videos obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1606 may include a display panel 16061, and the display panel 16061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1607 includes at least one of a touch panel 16071 and other input devices 16072. The touch panel 16071 is also called a touch screen. The touch panel 16071 may include two parts: a touch detection device and a touch controller. The other input devices 16072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here.

[0331] The memory 1609 can be used to store software programs and various data. The memory 1609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area may store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1609 may include a volatile memory or a non-volatile memory, or the memory 1609 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DRRAM). The memory 1609 in the embodiments of the present application includes, but is not limited to, these and any other suitable types of memories.

[0332] The processor 1610 may include one or more processing units; optionally, the processor 1610 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 1610 either.

[0333] The embodiments of the present application also provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above embodiment of the image processing method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0334] Among them, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media such as computer read-only memory ROM, random access memory RAM, magnetic disks, or optical discs, etc.

[0335] Another embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above embodiment of the image processing method and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0336] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0337] The embodiments of the present application provide a computer program product. The program product is stored in a storage medium and is executed by at least one processor to implement each process of the above embodiment of the image processing method and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0338] It should be noted that in this article, the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article, or device. Without more limitations, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article, or device including that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described method may be executed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0339] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0340] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.

Claims

1. An image processing method, characterized in that: Applied to a first electronic device, the method includes: Receiving a first parameter sent by a second electronic device, where the first parameter includes a first posture parameter of the second electronic device; Acquire a first picture based on the first posture parameter and a posture conversion parameter, wherein the posture conversion parameter is used to convert the posture parameter of the second electronic device; A first message is sent to the second electronic device, where the first message includes the first screen.

2. The method according to claim 1, characterized in that The acquiring a first picture based on the first posture parameter and the posture conversion parameter includes: Based on the first posture parameter and the posture conversion parameter, acquiring shooting parameters of a virtual camera of the first electronic device, wherein the virtual camera is used to acquire a virtual picture of the first electronic device; Based on the shooting parameters, acquiring a first space element in the virtual space of the first electronic device; The first spatial element is rendered to obtain the first picture.

3. The method according to claim 2, characterized in that The first parameters also include first camera projection parameters of the second electronic device, and the shooting parameters include a shooting position of the virtual camera, a shooting angle of the virtual camera, and second camera projection parameters of the virtual camera; The acquiring, based on the first posture parameter and the posture conversion parameter, shooting parameters of the virtual camera of the first electronic device includes: Based on the first posture parameter and the posture conversion parameter, obtaining a second posture parameter; Based on the second posture parameter, determining the shooting position and the shooting angle; The first camera projection parameter is determined as the second camera projection parameter.

4. The method according to claim 2, characterized in that: The first parameter also includes a screen resolution of the second electronic device; The rendering of the first spatial element to obtain the first picture includes: Based on the screen resolution, the first spatial element is rendered to obtain the first picture, and the resolution of the first picture is smaller than the screen resolution.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: A first marked image displayed by the second electronic device is identified and located to obtain the posture conversion parameter, wherein the first marked image is used to indicate the position of the second electronic device.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Identify and locate the second marker image at a preset position to obtain a third pose parameter; Acquire a fourth posture parameter, where the fourth posture parameter is obtained by the second electronic device identifying and locating the second marking image; Based on the third posture parameter and the fourth posture parameter, the posture conversion parameter is obtained.

7. An image processing method, characterized in that: Applied to a second electronic device, the method includes: Sending a first parameter of the second electronic device to the first electronic device, where the first parameter includes a first posture parameter of the second electronic device, and the first posture parameter is used by the first electronic device to acquire a picture; Receiving a first message sent by a first electronic device, wherein the first message includes a first screen; After receiving the plurality of first messages, a second screen is generated based on the plurality of first screens.

8. The method according to claim 7, characterized in that Each first message also includes a second posture parameter; The generating of the second picture based on the plurality of first pictures comprises: Based on the plurality of first images and the plurality of second pose parameters, a second image is generated.

9. The method according to claim 8, characterized in that The generating of the second picture based on the plurality of first pictures and the plurality of second posture parameters comprises: Determine, based on a second pose parameter corresponding to a first target picture and a second pose parameter corresponding to a second target picture, a first pixel point corresponding to each pixel point of the first target picture in the second target picture, wherein the first target picture is any picture of the multiple first pictures, and the second target picture is a picture of the multiple first pictures except the first target picture; Taking a weighted average of a first pixel point in the second target picture and a corresponding pixel point in the first target picture to obtain a third picture; Based on the screen resolution of the second electronic device, an interpolation operation is performed on the third picture to obtain the second picture.

10. An image processing device, characterized in that: The device comprises: A receiving module, configured to receive a first parameter sent by a second electronic device, wherein the first parameter includes a first posture parameter of the second electronic device; An acquisition module, configured to acquire a first picture based on the first posture parameter and a posture conversion parameter, wherein the posture conversion parameter is used to convert a posture parameter of the second electronic device; The sending module is used to send a first message to the second electronic device, where the first message includes the first screen.

11. The device according to claim 10, characterized in that The acquisition module is specifically used for: Based on the first posture parameter and the posture conversion parameter, acquiring a shooting parameter of a virtual camera of a first electronic device, wherein the virtual camera is used to shoot a virtual picture of the first electronic device; Based on the shooting parameters, acquiring a first space element in the virtual space of the first electronic device; The first spatial element is rendered to obtain the first picture.

12. The device according to claim 11, characterized in that The first parameters also include first camera projection parameters of the second electronic device, and the shooting parameters include a shooting position of the virtual camera, a shooting angle of the virtual camera, and second camera projection parameters of the virtual camera; The acquisition module is specifically used for: Based on the first posture parameter and the posture conversion parameter, obtaining a second posture parameter; Based on the second posture parameter, determining the shooting position and the shooting angle; The first camera projection parameter is determined as the second camera projection parameter.

13. The device according to claim 11, characterized in that The first parameter also includes a screen resolution of the second electronic device; The acquisition module is specifically used for: Based on the screen resolution, the first spatial element is rendered to obtain the first picture, and the resolution of the first picture is smaller than the screen resolution.

14. The device according to any one of claims 10 to 13, characterized in that The device also includes: The processing module is used to identify and locate the first marking image displayed by the second electronic device to obtain the posture conversion parameter, and the first marking image is used to indicate the position of the second electronic device.

15. The device according to any one of claims 10 to 13, characterized in that The device also includes: A processing module, used for identifying and locating the second marker image at a preset position to obtain a third posture parameter; Acquire a fourth posture parameter, where the fourth posture parameter is obtained by the second electronic device identifying and locating the second marking image; Based on the third posture parameter and the fourth posture parameter, the posture conversion parameter is obtained.

16. An image processing device, characterized in that: The device comprises: A sending module, used for sending a first parameter of a second electronic device to a first electronic device, wherein the first parameter includes a first posture parameter of the second electronic device, and the first posture parameter is used for the first electronic device to acquire a picture; A receiving module, configured to receive a first message sent by a first electronic device, wherein the first message includes a first picture; The processing module is used to generate a second picture based on the multiple first pictures after receiving the multiple first messages.

17. The device according to claim 16, characterized in that Each first message also includes a second posture parameter; Processing module, specifically used for: Based on the plurality of first images and the plurality of second pose parameters, a second image is generated.

18. The device according to claim 16, characterized in that The processing module is specifically used for: Determine, based on a second pose parameter corresponding to a first target picture and a second pose parameter corresponding to a second target picture, a first pixel point corresponding to each pixel point of the first target picture in the second target picture, wherein the first target picture is any picture of the multiple first pictures, and the second target picture is a picture of the multiple first pictures except the first target picture; Taking a weighted average of a first pixel point in the second target picture and a corresponding pixel point in the first target picture to obtain a third picture; Based on the screen resolution of the second electronic device, an interpolation operation is performed on the third picture to obtain the second picture.

19. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method as described in any one of claims 1 to 6 are implemented, or when the program or instruction is executed by the processor, the steps of the image processing method as described in any one of claims 7 to 9 are implemented.

20. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the image processing method as described in any one of claims 1 to 6 are implemented, or the steps of the image processing method as described in any one of claims 7 to 9 are implemented.

Images