Dual-picture display method, device and playing equipment

Abstract

This application relates to the field of image processing technology and provides a dual-screen display method, apparatus, and playback device, including: when a target operation is detected in the panoramic screen displayed on the panoramic screen playback client, determining the distance change corresponding to the target operation to obtain a panoramic distance change; determining the second screen coordinates of the panoramic screen based on the panoramic distance change and the first screen coordinates of the panoramic screen; determining the vertex coordinates of the panoramic screen based on the second screen coordinates of the panoramic screen and a preset screen vertex mapping relationship; rendering a texture map corresponding to the vertex coordinates of the panoramic screen; and displaying the rendered screen through the panoramic screen playback client.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to dual-screen display methods, apparatus, playback devices and computer-readable storage media. Background Technology

[0002] With the development of network technology, people can use panoramic close-up dual-view shooting systems to monitor specific scenes.

[0003] like Figure 1 As shown, this panoramic close-up dual-view shooting system can be composed of a network camera (IP Camera, IPC) and a playback device. The playback device includes a panoramic view playback client, a close-up view playback client, and a viewfinder. The viewfinder is included in the panoramic view playback client and can be adjusted within the display range of the panoramic monitoring screen. After adjustment, the close-up view playback client will magnify and display the image corresponding to the view captured by the viewfinder. Specifically, the single-channel video stream acquired by the IPC is decoded, and the resulting texture map is rendered twice (one rendering renders the entire texture map, and the other rendering renders only the portion of the texture map corresponding to the view captured by the viewfinder). The rendering results from the two renderings are then projected onto the viewports of the panoramic view playback client and the close-up view playback client for display, respectively.

[0004] In existing panoramic close-up dual-view shooting systems, although the close-up view playback client can display the corresponding image according to the adjustment of the viewfinder position, the content that the panoramic view playback client can display is still difficult to meet the needs of users. Summary of the Invention

[0005] This application provides a dual-screen display method, apparatus, playback device, and computer-readable storage medium, which can solve the problem that the screen content displayed by existing panoramic screen playback clients cannot meet the needs of users.

[0006] In a first aspect, embodiments of this application provide a dual-screen display method, applied to a playback device including a panoramic screen playback client, the dual-screen display method comprising:

[0007] When a target operation is detected in the panoramic image displayed on the panoramic image playback client, the distance change corresponding to the target operation is determined to obtain the panoramic distance change. The panoramic image is obtained by rendering the projection information of the texture map projected into the viewport. The target operation includes translation operation and / or scaling operation.

[0008] The second screen coordinates of the panoramic image are determined based on the change in panoramic distance and the first screen coordinates of the panoramic image. The first screen coordinates of the panoramic image are the screen coordinates of the panoramic image before the target operation is detected. The screen coordinates are the coordinates in the screen coordinate system established by the screen of the panoramic image playback client. The second screen coordinates of the panoramic image are the screen coordinates of the panoramic image after the target operation is detected.

[0009] The vertex coordinates of the panoramic image are determined based on the second screen coordinates of the panoramic image and a preset screen vertex mapping relationship. The vertex coordinates are coordinates in the vertex coordinate system established with the viewport, and the preset screen vertex mapping relationship is used to reflect the mapping relationship between the screen coordinate system and the vertex coordinate system.

[0010] Render the texture map corresponding to the vertex coordinates of the panoramic image;

[0011] The panoramic view is displayed and rendered by the client.

[0012] Secondly, embodiments of this application provide a dual-screen display device, applied to a playback device including a panoramic screen playback client, comprising:

[0013] A panoramic distance change determination module is used to determine the distance change corresponding to the target operation when the panoramic image displayed by the panoramic image playback client detects the target operation, and obtain the panoramic distance change. The panoramic image is obtained by rendering the projection information of the texture map projected into the viewport. The target operation includes translation operation and / or scaling operation.

[0014] The second screen coordinate determination module for the panoramic image is used to determine the second screen coordinate of the panoramic image based on the panoramic distance change and the first screen coordinate of the panoramic image. The first screen coordinate of the panoramic image is the screen coordinate of the panoramic image before the target operation is detected. The screen coordinate is the coordinate under the screen coordinate system established with the screen of the panoramic image playback client. The second screen coordinate of the panoramic image is the screen coordinate of the panoramic image after the target operation is detected.

[0015] A vertex coordinate determination module for panoramic images is used to determine the vertex coordinates of the panoramic images based on the second screen coordinates of the panoramic images and a preset screen vertex mapping relationship. The vertex coordinates are coordinates in a vertex coordinate system established with the viewport, and the preset screen vertex mapping relationship is used to reflect the mapping relationship between the screen coordinate system and the vertex coordinate system.

[0016] The texture mapping rendering module is used to render texture maps corresponding to the vertex coordinates of the panoramic image;

[0017] The screen display module is used to display the rendered screen through the panoramic screen playback client.

[0018] Thirdly, embodiments of this application provide a playback device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0020] Fifthly, embodiments of this application provide a computer program product that, when run on a playback device, causes the playback device to execute the method described in the first aspect above.

[0021] The beneficial effects of the embodiments in this application compared with the prior art are:

[0022] Since the target operation includes translation and / or zooming, and both translation and zooming operations result in a certain distance change, if a target operation is detected in the panoramic view displayed on the client side, the distance change corresponding to that target operation can be determined, thus obtaining the panoramic distance change. The first screen coordinates of the panoramic view are the screen coordinates corresponding to the panoramic view before the target operation is detected, and the second screen coordinates are the screen coordinates corresponding to the panoramic view after the target operation is detected. Therefore, the second screen coordinates of the panoramic view can be determined based on the panoramic distance change and the first screen coordinates. Furthermore, since the panoramic image is rendered by projecting texture maps onto the viewport, and the preset screen vertex mapping relationship reflects the mapping relationship between the screen coordinate system and the aforementioned vertex coordinate system, the vertex coordinates of the panoramic image can be determined based on the second screen coordinates of the panoramic image and the preset screen vertex mapping relationship. This means determining the coordinates of the panoramic image in the vertex coordinate system established with the viewport, which is equivalent to determining the range of the texture map to be rendered. Therefore, after rendering the texture map corresponding to the vertex coordinates of the panoramic image, the rendered image can be displayed through the panoramic image playback client, i.e., displaying an image that matches the change in panoramic distance. In other words, in this embodiment, by changing the vertex positions corresponding to the texture map, the range of the image displayed on the screen of the panoramic image playback client changes, thereby increasing the flexibility of the image displayed by the panoramic image playback client and increasing the probability that the image displayed by the panoramic image playback client meets the user's needs. Furthermore, since the display range of the panoramic image is adjustable, when the framing range corresponding to the viewfinder is small, the panoramic image can be magnified to make the viewfinder magnify synchronously. In this way, the user can more accurately judge whether the framing range of the viewfinder is the range that the user actually wants to select based on the magnified viewfinder, thereby improving the accuracy of the framing range. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0024] Figure 1 This is a structural diagram of a panoramic close-up dual-view shooting system;

[0025] Figure 2 This is a schematic diagram showing the correspondence between texture maps, viewport range, and screen display images provided in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the screen display when the vertex coordinates of the texture map exceed the viewport range, according to an embodiment of this application.

[0027] Figure 4 This is a flowchart illustrating a dual-screen display method according to an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of a normalized screen coordinate system established with the upper left corner of the panoramic image playback client screen as the origin, provided in an embodiment of this application.

[0029] Figure 6 This is an embodiment provided in this application. Figure 5 A schematic diagram showing the screen coordinate system superimposed with the vertex coordinate system;

[0030] Figure 7 This is a schematic diagram of another screen coordinate system superimposed on a vertex coordinate system, provided in another embodiment of this application;

[0031] Figure 8 This is a schematic diagram illustrating the relative positional relationship between a viewfinder and a panoramic viewport, as provided in an embodiment of this application.

[0032] Figure 9 This is a schematic diagram illustrating how to correct the viewfinder position after performing target operations on a panoramic image, as provided in an embodiment of this application.

[0033] Figure 10 This is a schematic diagram illustrating how to control the close-up image in conjunction with the viewfinder when performing a target operation on the viewfinder, as provided in an embodiment of this application.

[0034] Figure 11 This is a schematic diagram illustrating how the viewfinder is linked when performing target operations on a close-up image, as provided in an embodiment of this application.

[0035] Figure 12 This is a schematic diagram of the structure of a dual-screen display device provided in an embodiment of this application;

[0036] Figure 13 This is a schematic diagram of the playback device provided in the embodiments of this application. Detailed Implementation

[0037] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0038] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0039] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0040] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0041] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0042] In a panoramic-close-up dual-view shooting system, the view displayed on the close-up view playback client can be adjusted by changing the position of the viewfinder on the panoramic view playback client. For example, by panning the viewfinder to change its position on the panoramic view playback client, the panoramic view captured by the viewfinder before and after the panning (which is the view displayed on the panoramic view playback client) will be different, and therefore, the view displayed on the close-up view playback client will also change accordingly.

[0043] While close-up view playback clients can display corresponding images based on viewfinder adjustments, panoramic views are rendered by projecting the entire texture map onto the viewport. Therefore, the panoramic view displayed by these clients is fixed and cannot be adjusted, making it difficult to meet user needs. Furthermore, when the panoramic view is fixed and the target framing area is small, it's difficult to accurately select the corresponding target area through the viewfinder, further complicating the display of content from close-up view playback clients.

[0044] Before introducing the dual-screen display method provided in the embodiments of this application, the principle of dual-screen display will be briefly introduced below.

[0045] After the IPC captures the video stream, it decodes the single video stream captured by the IPC and uses the original complete image obtained from decoding the video stream as the texture map of the graphics rendering program. That is, the texture map contains the complete original graphics data.

[0046] The area of ​​the graphics displayed in the playback client is controlled by the viewport of the graphics rendering program. This viewport is a rectangular area within the window, typically coinciding with the window. During rendering, texture maps are projected onto the viewport to generate the final displayable image. Texture maps not projected into the viewport (i.e., graphics outside the viewport area) will not be rendered. It's important to note that when two playback clients exist, such as a panoramic playback client and a close-up playback client, these two clients correspond to different viewports, which can be distinguished as a panoramic viewport and a close-up viewport.

[0047] When projecting a texture map onto the viewport and rendering it onto the screen of the playback client, three main coordinate systems are involved: texture coordinate system, vertex coordinate system, and screen coordinate system.

[0048] The texture coordinate system is a coordinate system constructed based on the texture map. The origin of this coordinate system can be set at the lower left corner of the texture map, and the x-axis points horizontally to the right and vertically upwards, respectively. Typically, the coordinate values ​​of the texture data (i.e., texture coordinates) can be set to all be within the range of [0, +1].

[0049] The vertex coordinate system is a coordinate system constructed based on the viewport. The origin of this vertex coordinate system can be set at the center of the viewport. Typically, all coordinate values ​​(i.e., vertex coordinates) within the viewport range are within the range of [-1, +1]. Points outside this range will be ignored by the graphics rendering program.

[0050] The screen coordinate system is a coordinate system constructed based on the screen of the client playing the video. The origin of the screen coordinate system can be set at the upper left corner of the screen, and the x-axis and y-axis are horizontal to the right and vertical to the bottom, respectively.

[0051] The graphics rendering program maps texture coordinates to vertex coordinates, and then maps vertex coordinates within the viewport to screen coordinates. The vertex positions after texture mapping can be described by the x, y, width, and height coordinates of the bottom-left vertex, which by default is (-1, -1, 2, 2). This corresponds to the coordinates of the four vertices as (-1, -1), (-1, 1), (1, -1), and (1, 1), meaning the range of vertex coordinates overlaps with the viewport range, allowing the screen to display the complete image corresponding to the texture map. Figure 2 As shown.

[0052] However, when the vertex coordinates of the texture map are not within the viewport range, such as Figure 3As shown, assuming the vertex position is (-2, -2, 4, 4), its corresponding vertex coordinates are (-2, -2), (-2, 2), (2, -2), (2, 2). At this time, the texture map will be enlarged and exceed the viewport range, but only the local scene within the viewport range will be rendered on the screen. That is, the screen display range can be adjusted by changing the vertex position corresponding to the texture map.

[0053] The following describes how to adjust the screen display range of the panoramic video playback client by changing the vertex positions corresponding to the texture map, thereby improving the flexibility of the displayed image.

[0054] Figure 4 A flowchart illustrating a dual-screen display method according to an embodiment of this application is shown. This dual-screen display method is applied to a playback device including a panoramic screen playback client, and is described in detail below:

[0055] S41, when a target operation is detected in the panoramic image displayed on the panoramic image playback client, the distance change corresponding to the target operation is determined, and the panoramic distance change is obtained. The panoramic image is obtained by rendering the projection information of the texture map projected into the viewport. The target operation includes translation operation and / or scaling operation.

[0056] Specifically, when the target operation is a translation operation, the change may be in the x-coordinate, the y-coordinate, or both, but the overall size of the panoramic image remains unchanged. That is, the change in panoramic distance corresponding to a translation operation includes the changes in the x-coordinate and / or y-coordinate. When the target operation is a zoom operation, only the x-coordinate and / or y-coordinate change, and the overall size of the panoramic image also changes. For example, when the target operation is a zoom-in operation, the overall size of the panoramic image will increase. That is, the change in panoramic distance corresponding to a zoom operation includes the change in size, as well as the changes in the x-coordinate and / or y-coordinate.

[0057] In this embodiment, the coordinates of the icon or object that issued the target operation when the target operation was first detected (assumed to be the first coordinates) are determined, and the coordinates of the icon or object that issued the target operation when the target operation was last detected (assumed to be the second coordinates) are determined. The aforementioned panoramic distance change is determined based on the first and second coordinates. This panoramic distance change is the distance change corresponding to the screen coordinate system.

[0058] For example, when the target operation is a translation operation, and the object issuing the target operation is the user's finger, the aforementioned panoramic distance change is determined by detecting the movement distance of the user's finger. For example, assuming the coordinates A when the user's finger is first detected are (x1, y1), and the coordinates B when the user's finger is last detected are (x2, y1), then the distance change corresponding to this translation operation is (x2 - x1).

[0059] S42, determine the second screen coordinates of the panoramic image based on the panoramic distance change and the first screen coordinates of the panoramic image, wherein the first screen coordinates of the panoramic image are the screen coordinates of the panoramic image before the target operation is detected, the screen coordinates are the coordinates in the screen coordinate system established by the screen of the panoramic image playback client, and the second screen coordinates of the panoramic image are the screen coordinates of the panoramic image after the target operation is detected.

[0060] The first and second screen coordinates of the panoramic view mentioned above both include horizontal and vertical coordinates. The first screen coordinates indicate the extent of the panoramic view before the target operation is detected. For example, when the screen of the panoramic view playback client is rectangular, the first screen coordinates include at least four coordinates, that is, the screen coordinates corresponding to the four vertices of the rectangle. Similarly, the second screen coordinates indicate the extent of the panoramic view after the target operation is detected, and the number of second screen coordinates is equal to the number of first screen coordinates.

[0061] Specifically, when the change in panoramic distance includes the change in the horizontal coordinate, the horizontal coordinate in the second screen coordinate of the panoramic image is calculated by adding or subtracting the change in the horizontal coordinate with the horizontal coordinate of the first screen coordinate of the panoramic image. Similarly, when the change in panoramic distance includes the change in the vertical coordinate, the vertical coordinate in the second screen coordinate of the panoramic image is calculated by adding or subtracting the change in the vertical coordinate with the vertical coordinate of the first screen coordinate of the panoramic image. Whether an addition or subtraction operation is required is determined by the direction of the target operation. For example, when the translation operation is to the right, the horizontal coordinate may be set to require addition; conversely, when the translation operation is to the down, the vertical coordinate may be set to require addition.

[0062] S43, determine the vertex coordinates of the panoramic image based on the second screen coordinates of the panoramic image and the preset screen vertex mapping relationship, wherein the vertex coordinates are coordinates in the vertex coordinate system established with the viewport, and the preset screen vertex mapping relationship is used to reflect the mapping relationship between the screen coordinate system and the vertex coordinate system.

[0063] In this embodiment of the application, when the screen and viewport of the panoramic image playback client are determined, the mapping relationship between the screen and the viewport is essentially fixed, that is, the above-mentioned screen vertex mapping relationship is essentially fixed. Therefore, the screen vertex mapping relationship can be determined in advance.

[0064] Alternatively, the above screen vertex mapping relationship can be determined in the following way:

[0065] Establish a system with the top left corner of the panoramic playback client screen as the origin. Figure 5 The normalized screen coordinate system shown. Figure 5 In this model, the length and width of the screen are taken as one unit length on the horizontal and vertical axes, respectively. It is assumed that the length and width of the viewfinder are ratioW and ratioH times the length and width of the screen. When ratioW is equal to ratioH, it indicates that the length and width of the viewfinder are in equal proportions. Otherwise, it indicates that the length and width of the viewfinder are not in equal proportions.

[0066] Assuming the coordinates of the top-left corner of the viewfinder are (x, y), the relative position of the viewfinder and the panoramic viewport can be represented by (x, y, ratioW, ratioH). Assuming the coordinate range of the screen is the same as the coordinate range of the texture map, (x, y, ratioW, ratioH) can also describe the relative position of the viewfinder and the texture map, with the position parameter values ​​ranging from [0, 1].

[0067] exist Figure 5 After superimposing the vertex coordinate system in the viewfinder, we obtain Figure 6 In vertex coordinates, the viewfinder can be considered the viewport of the close-up image rendering program (i.e., the close-up viewport), and the complete texture map displayed by the panoramic viewport is also the texture map of the close-up image. The image within the viewfinder viewport (i.e., the close-up viewport) will be displayed on the screen of the close-up image playback client. Figure 6 In the vertex coordinate system, the top left corner of the viewfinder is fixed at (-1, 1), the bottom left corner is (-1, -1), and the side length is 2.

[0068] Will Figure 6 Processed as follows Figure 7 The form shown, from Figure 7 It can be seen that x and y are normalized coordinates with values ​​of [0,1]. L1 = x, and L2 + L3 + L4 = 1. L2 = y, L3 = ratioH, then L4 = 1 - y - ratioH.

[0069] Because x vertex ,y vertex All are less than 0, and -x vertex =J1+J2, -y vertex =J3+J4, that is, J1=-xvertex -1, J4 = -y vertex -1.

[0070] And because Right now Similarly,

[0071] For both screen coordinates and vertex coordinates, the same parameter corresponds to the same proportion for the width of the outer rectangle, and the same proportion for the height (or length) of the outer rectangle, that is:

[0072] Substituting the specific values, we get:

[0073] After simplification, we get:

[0074] That is, the screen vertex mapping relationship is:

[0075]

[0076] By performing an equation transformation on the above formula (1), we obtain the following formula (2):

[0077]

[0078] After obtaining the above screen vertex mapping relationship, substitute the second screen coordinates of the panoramic image into (x, y) in the above formula (1) to obtain the vertex coordinates (x, y) of the panoramic image. vertex ,y vertex It should be noted that since the vertex coordinates of the panoramic image are coordinates in the viewport coordinate system, and the second screen coordinates of the panoramic image mentioned above can indicate the range of the panoramic image after the target operation is detected, the vertex coordinates of the panoramic image calculated based on the second screen coordinates of the panoramic image mentioned above can also indicate the range of texture maps that need to be rendered.

[0079] S44, render the texture map corresponding to the vertex coordinates of the aforementioned panoramic image.

[0080] Once the vertex coordinates of the panoramic image are determined, the area of ​​the texture map to be rendered is also determined. Therefore, the area of ​​the texture map indicated by the vertex coordinates of the panoramic image can be rendered.

[0081] S45, the rendered image is displayed on the client through the aforementioned panoramic view playback.

[0082] Specifically, the rendered image is displayed at the second screen coordinates of the panoramic view.

[0083] Since the target operation includes translation and / or zooming, and both translation and zooming operations result in a certain distance change, if a target operation is detected in the panoramic view displayed on the client side, the distance change corresponding to that target operation can be determined, thus obtaining the panoramic distance change. The first screen coordinates of the panoramic view are the screen coordinates corresponding to the panoramic view before the target operation is detected, and the second screen coordinates are the screen coordinates corresponding to the panoramic view after the target operation is detected. Therefore, the second screen coordinates of the panoramic view can be determined based on the panoramic distance change and the first screen coordinates. Furthermore, since the panoramic image is rendered by projecting texture maps onto the viewport, and the preset screen vertex mapping relationship reflects the mapping relationship between the screen coordinate system and the aforementioned vertex coordinate system, the vertex coordinates of the panoramic image can be determined based on the second screen coordinates of the panoramic image and the preset screen vertex mapping relationship. This means determining the coordinates of the panoramic image in the vertex coordinate system established with the viewport, which is equivalent to determining the range of the texture map to be rendered. Therefore, after rendering the texture map corresponding to the vertex coordinates of the panoramic image, the rendered image can be displayed through the panoramic image playback client, i.e., displaying an image that matches the change in panoramic distance. In other words, in this embodiment, by changing the vertex positions corresponding to the texture map, the range of the image displayed on the screen of the panoramic image playback client changes (i.e., the range of the image displayed on the screen of the panoramic image playback client is no longer fixed), thereby increasing the flexibility of the image displayed by the panoramic image playback client and increasing the probability that the image displayed by the panoramic image playback client meets the user's needs. Furthermore, since the display range of the panoramic image is adjustable, when the framing range corresponding to the viewfinder is small, the panoramic image can be magnified to make the viewfinder magnify synchronously. In this way, the user can more accurately judge whether the framing range of the viewfinder is the range that the user actually wants to select based on the magnified viewfinder, thereby improving the accuracy of the framing range.

[0084] The above describes how the panoramic view displayed on the panoramic playback client changes according to the target operation detected on the panoramic view (i.e., the panoramic view outside the area contained by the viewfinder). When the panoramic playback client includes a viewfinder, if the panoramic view displayed on the screen changes, the screen coordinates of the viewfinder within the panoramic playback client may or may not change.

[0085] Optionally, considering that when the panoramic view displayed on the screen of the panoramic view playback client changes, if the screen coordinates of the viewfinder in the panoramic view playback client remain unchanged, the position of the viewfinder displayed will not match the actual viewing area in the texture map. Therefore, in order to match the position of the viewfinder displayed with the actual viewing area in the texture map, the screen coordinates of the viewfinder on the screen need to be updated synchronously during the change of vertex coordinates in the texture map. That is, in S45 above, the dual-screen display method provided in this application embodiment further includes:

[0086] The viewfinder is corrected so that the first relative position and the second relative position remain unchanged. The first relative position is the relative position of the viewfinder's framing area on the texture map and the texture map before the target operation is detected. The second relative position is the corrected relative position of the viewfinder's framing area on the texture map and the texture map.

[0087] Specifically, the screen coordinates of the viewfinder relative to the panoramic viewport are determined before the target operation is detected. Since the relative position of the viewfinder to the panoramic viewport screen and the relative position of the framing area to the texture map are the same, the relative position of the framing area to the texture map can be determined based on the relative position of the viewfinder to the panoramic viewport screen. When the relative position of the viewfinder to the panoramic viewport screen is the same as the relative position before the target operation was detected, the aforementioned first relative position can be determined based on this relative position. Thus, when the panoramic image playback client adjusts its displayed panoramic image according to the detected target operation, the viewfinder, which adjusts with the panoramic image, is corrected so that the relative position of the framing area to the texture map (i.e., the second relative position) corresponding to the corrected viewfinder remains unchanged from the first relative position.

[0088] In some embodiments, considering that when the panoramic view displayed on the screen of the panoramic playback client changes (such as by translation or scaling), the viewfinder located on the screen will also change, such as by changing its screen coordinates, or by changing both its screen coordinates and size. Therefore, when correcting the viewfinder, it is necessary to consider the screen coordinates, or the screen coordinates and size. In this case, the above-mentioned viewfinder correction includes:

[0089] The position of the viewfinder is corrected, and the position of the viewfinder includes screen coordinates, or includes screen coordinates and size.

[0090] When correcting the viewfinder's screen coordinates, the corrected screen coordinates are first calculated, and then the viewfinder's current screen coordinates are corrected to the calculated corrected screen coordinates. Specifically, the viewfinder's screen coordinates are determined based on the viewfinder's first texture coordinates, the vertex coordinates of the panoramic image, and a preset screen texture mapping relationship. The viewfinder's first texture coordinates are the texture coordinates of the viewfinder before the target operation is detected (i.e., the texture coordinates corresponding to the framing area). The texture coordinates are coordinates in a texture coordinate system established using the texture map. The preset screen texture mapping relationship reflects the mapping relationship between the screen coordinate system and the texture coordinate system. It should be noted that once the viewfinder's position on the screen is fixed, the screen coordinates in the screen coordinate system and the texture coordinate system (such as the viewfinder's first texture coordinates) can be calculated based on the mapping relationship between the screen coordinate system and the texture coordinate system.

[0091] When correcting the size of the viewfinder, the corrected size of the viewfinder can be calculated in the following way: based on the ratio between the size of the viewfinder and the screen size of the panoramic image playback client, and the ratio between the size of the texture map and the screen size of the panoramic image playback client, the new size of the viewfinder is calculated.

[0092] The preset screen texture mapping relationship can be determined through the following relationship:

[0093] Once the size of the viewfinder on the screen is determined, the relative position of the viewfinder's framing area and the texture map is also determined; this relative position will remain unchanged for the time being. For a description of the set relative relationship between the viewfinder and the screen, refer to... Figure 8 A normalized screen coordinate system is established with the top-left corner of the texture map. Assuming the normalized coordinates of the top-left corner of the viewfinder are (areaX, areaY), and the width and height of the viewfinder are ratioW and ratioH times the width and height of the texture map, then the relative position of the viewfinder on the texture map is (areaX, areaY, ratioW, ratioH). This position includes information about the coordinates (areaX, areaY) and dimensions (ratioW, ratioH).

[0094] A vertex coordinate system is established based on the panoramic viewport. Assuming the coordinate values ​​of the viewport's vertex coordinate system are fixed within the range of [-1, 1], the side length of the panoramic viewport is taken as 2 units. Figure 8 In the middle, the bottom left corner of the texture map has vertex coordinates (x, y, y) in the vertex coordinate system. vertex ,y vertex If the width of the texture map is sideW and the height is sideH, then (x) can be used.vertex ,y vertex The coordinates (sideW, sideH) describe the position of the texture map in the vertex coordinate system. This position is determined each time the panoramic view is adjusted. When the side length of the panoramic view is M units, the scaling factors of the texture map relative to the width and height of the panoramic view are sideW / M and sideH / M times, respectively. For example, when M is 2, sideW / M and sideH / M are sideW / 2 and sideH / 2, respectively. Of course, if the panoramic view is scaled proportionally, sideW / M and sideH / M are equal.

[0095] The normalized coordinates of the top-left corner of the viewfinder are (areaX, areaY), and the normalized coordinates of the bottom-left corner of the texture map are (0, 1). Their relative coordinates are (areaX, areaY-1). A similarly normalized texture coordinate system is established at the bottom-left corner of the texture map, with the y-axis of the two systems opposite. Therefore, the texture coordinates of the top-left corner of the viewfinder are (areaX, 1-areaY). Combining the scaling factors of the texture map relative to the panoramic viewport's width and height, when converting the texture coordinates to vertex coordinates, the x and y coordinates are multiplied by the coefficients sideW / 2 and sideH / 2 respectively. This gives the coordinates of the top-left corner of the viewfinder in the panoramic viewport vertex coordinate system as (sideW*areaX / 2, sideH*(1-areaY) / 2).

[0096] Assuming Figure 8 If the viewport and texture map in the image are considered as the viewfinder and the screen respectively, then... Figure 8 The normalized coordinates (x′, y′) of the top-left corner of the viewport correspond to the screen coordinates (x, y) of the top-left corner of the viewfinder. Figure 8 The aspect ratios 2 / sideW and 2 / sideH of the center viewport relative to the texture map correspond to the aspect ratios ratioW and ratioH of the viewfinder and the screen. Substituting these into formula (2), we get:

[0097]

[0098] Combining the coordinate difference between the top left corners of the viewfinder and the panoramic viewport in the normalized coordinate system, the screen coordinates of the top left corner of the viewfinder relative to the top left corner of the panoramic viewport can be calculated. Combining formulas (1) and (3), we obtain:

[0099]

[0100] Obtain the pixel width and height of the panoramic viewport's screen (i.e., the pixel width and height of the panoramic viewport). Assuming the pixel width and height are width and height respectively, the pixel width and height of the texture map are... sideH*height / 2. Using this as a coefficient in formula (4), we obtain the screen coordinates of the viewfinder in the panoramic viewport:

[0101]

[0102] The viewfinder's pixel width and height are:

[0103]

[0104] Formula (5) above is a preset screen texture mapping relationship. In formula (5), areaX and areaY are the first texture coordinates of the viewfinder. In formula (5), x vertex ,y vertex The coordinates of the vertex of the panoramic image are given. Since the width and height of the texture map are fixed (i.e., sideW and sideH are fixed), and the pixel width and height of the screen are also fixed (i.e., width and height are fixed), the screen coordinates of the viewfinder (x, y, y) can be calculated by substituting the first texture coordinates of the viewfinder and the vertex coordinates of the panoramic image into formula (5). ui y ui ).

[0105] Formula (6) above is used to correct the size of the viewfinder. In formula (6), ratioW and ratioH are the proportional relationships between the size of the viewfinder and the screen size of the panoramic image playback client. `sideH*height / 2` represents the ratio between the size of the texture map and the screen size of the panoramic playback client. That is, based on the ratios between the viewfinder size and the panoramic playback client's screen size, and the ratio between the texture map size and the panoramic playback client's screen size, the corrected viewfinder size can be determined.

[0106] To more clearly illustrate the differences before and after viewfinder correction, the following will combine... Figure 9 Describe it.

[0107] exist Figure 9 In the process, when a zoom or pan operation is detected in the panoramic view, it is equivalent to the texture map being scaled or panned. At this time, the viewfinder needs to be corrected to ensure that the viewfinder displayed on the screen of the panoramic view playback client or the close-up view playback client is in the framing area corresponding to the texture map, and its relative position to the texture map remains unchanged before and after the target operation on the panoramic view.

[0108] In some embodiments, the playback device of this application further includes a close-up view playback client, which is used to magnify and display the panoramic view captured by the viewfinder. To distinguish it from the panoramic view, the view displayed by the close-up view playback client is referred to as a close-up view. In this application embodiment, the content of the view displayed by the close-up view playback client can also be adjusted by directly operating the viewfinder. In this case, the dual-view display method provided by this application embodiment further includes:

[0109] A1. When the viewfinder detects a target operation, determine the distance change corresponding to the target operation to obtain the viewfinder distance change.

[0110] Specifically, the target operation is detected in the viewfinder, including the target operation being detected on the image captured by the viewfinder, and also the target operation being detected in the viewfinder controls themselves (such as on the display frame corresponding to the viewfinder).

[0111] In this embodiment, when the target operation is a panning operation, the change in viewing distance corresponding to the panning operation includes the change in the horizontal and / or vertical coordinates. When the target operation is a zooming operation, the change in viewing distance corresponding to the zooming operation includes the change in size, and also includes the change in the horizontal and / or vertical coordinates. Since the process of determining the change in viewfinder distance is similar to the process of determining the change in panoramic distance, it will not be described again here.

[0112] Optionally, the coordinate system corresponding to the aforementioned viewfinder distance change can be either the screen coordinate system or the texture coordinate system.

[0113] A2. Update the viewfinder based on the change in viewfinder distance and the first position of the viewfinder, wherein the first position of the viewfinder is the position of the viewfinder before the target operation is detected, and the position includes the coordinate information and / or size of the viewfinder.

[0114] Optionally, when the coordinate system corresponding to the viewfinder distance change and the viewfinder's first position is different, it needs to be converted to the same coordinate system before performing the corresponding calculations. For example, when the viewfinder distance change is the change in the screen coordinate system, and the viewfinder's first position is the viewfinder's coordinate in the texture coordinate system before detecting the target operation, the viewfinder distance change can be converted from the screen coordinate system to the texture coordinate system according to the preset screen texture mapping relationship (i.e., the mapping relationship between the screen coordinate system and the texture coordinate system), to obtain the change in the texture coordinate system. Then, based on the change in the texture coordinate system and the viewfinder's first position, the viewfinder's coordinates in the texture coordinate system after detecting the target operation (assumed to be the viewfinder's second position) are determined. Then, the viewfinder displayed on the panoramic playback client is updated based on the viewfinder's second position. For example, converting the viewfinder's second position to the screen coordinate system, the converted position is the updated viewfinder position. Of course, if the change in viewfinder distance is a change in texture coordinates, then there is no need to transform the coordinate system for the change in viewfinder distance; it can be calculated directly.

[0115] Optionally, when the viewfinder distance change and the coordinate system corresponding to the viewfinder's first position are the same, no coordinate system transformation is required. For example, when the viewfinder distance change is a change in the screen coordinate system, and the viewfinder's first position is the viewfinder's coordinates in the screen coordinate system before the target operation was detected, the updated position of the viewfinder in the screen coordinate system can be directly calculated based on the viewfinder distance change and the viewfinder's first position. Alternatively, the viewfinder distance change can be transformed from the screen coordinate system to the texture coordinate system to obtain the change in the texture coordinate system, and the viewfinder's first position can be transformed from the screen coordinate system to the texture coordinate system to obtain the viewfinder's third position. Then, based on the change in the texture coordinate system and the viewfinder's third position, the viewfinder's coordinates in the texture coordinate system after the target operation was detected (assumed to be the viewfinder's second position) can be calculated. Finally, the viewfinder can be updated based on the viewfinder's second position.

[0116] In this embodiment, considering that performing a target operation on the viewfinder directly affects the screen coordinates and size of the viewfinder in the panoramic image playback client, the position of the viewfinder on the screen of the panoramic image playback client can be updated in real time. The coordinate differences caused by translation and the side length ratio differences caused by scaling are recorded during the operation. The position of the viewfinder after the operation is calculated by combining the viewfinder's position before the operation. For example, assuming the viewfinder's position in the texture coordinate system before the operation (i.e., the first position of the viewfinder) is (areaX, areaY, ratioW, ratioH), the horizontal and vertical translation components caused by translation during the operation are accumulated as Δx′ and Δy′, respectively, and the width and height ratio differences caused by scaling are multiplied as ΔratioW and ΔratioH, then the viewfinder position in the texture map after the operation (i.e., the second position of the viewfinder) is (areaX+Δx′, areaY+Δy′, ratioW*ΔratioW, ratioH*ΔratioH).

[0117] A3. Update the image displayed on the client side of the above close-up image playback according to the updated viewfinder position.

[0118] Specifically, considering that the image displayed by the close-up view playback client is related to the vertex coordinate range of the texture map projected into the close-up viewport, the vertex coordinates of the viewfinder in the framing area corresponding to the texture map after the target operation can be calculated by substituting the updated viewfinder position (assuming it is the second position of the viewfinder) into formula (1), rendering the texture map within these vertex coordinate ranges, and displaying the rendered image through the close-up view playback client.

[0119] In this embodiment, since target operations can be performed on the close-up screen in the close-up screen playback client, and the playback device updates the position of the viewfinder displayed in the panoramic screen playback client and the screen displayed in the close-up screen playback client after detecting the target operation, the number of playback clients that can perform target operations (not only in the panoramic screen but also in the close-up screen) is increased, thereby improving the user experience.

[0120] To more clearly describe the process of controlling the close-up image when performing target operations through the viewfinder, the following will combine... Figure 10 Describe it.

[0121] exist Figure 10 In the context of texture mapping, the close-up viewport is used to select the graphic range on which the texture mapping is projected.

[0122] When a user performs a pan (or zoom) operation on the viewfinder on the panoramic playback client's screen, the playback device calculates the change in viewfinder distance corresponding to the pan (or zoom) operation. Based on this change in viewfinder distance and the viewfinder's initial position (i.e., its position before the user performed the pan (or zoom) operation), the device determines the viewfinder's position after the operation and updates the viewfinder accordingly. Simultaneously, based on this position, the device determines the texture vertex position within the close-up viewport (i.e., the vertex coordinates corresponding to the viewfinder) and renders the corresponding image based on this vertex position, thus updating the image displayed on the close-up playback client.

[0123] In some embodiments, the position of the viewfinder in the panoramic view playback client can be controlled by performing target operations on the screen displayed by the close-up view playback client. In this case, the dual-screen display method provided in this application embodiment further includes:

[0124] B1. If a target operation is detected in the close-up screen displayed by the client, the distance change corresponding to the target operation is determined, and the close-up distance change is obtained.

[0125] The process of determining the change in close-up distance is similar to that of determining the change in panoramic distance, and will not be repeated here.

[0126] B2. Determine the second vertex position of the close-up image based on the aforementioned change in close-up distance and the first vertex position of the close-up image. The first vertex position of the close-up image is the vertex position of the close-up image before the aforementioned target operation is detected, and the second vertex position of the close-up image is the vertex position of the close-up image after the aforementioned target operation is detected. Both the first vertex position and the second vertex position of the close-up image include vertex coordinates and dimensions.

[0127] Specifically, considering that the target operation directly affects the close-up view, the close-up view can be updated by updating the vertex positions of the texture map. For example, suppose that before the target operation is detected, the vertex positions of the texture map in the close-up viewport are (vertexX,vertexY,sideW,sideH) (i.e., the first vertex positions of the close-up view), where the first two parameters represent the vertex coordinates and the last two parameters represent the dimensions (i.e., width and height). Suppose that the horizontal and vertical translation components caused by the translation operation are accumulated as Δx″ and Δy″, respectively, and the width and height ratio differences caused by the scaling operation are accumulated as ΔratioW and ΔratioH, then the vertex positions of the texture map after the target operation are (vertexX+Δx″,vertexY+Δy″,sideW*ΔratioW,sideH*ΔratioH) (i.e., the second vertex positions of the close-up view).

[0128] B3. Render the texture map corresponding to the second vertex position in the aforementioned close-up shot.

[0129] Specifically, the second vertex position of the close-up image is applied to the graphics rendering program in the close-up image, that is, the texture image in the close-up viewport is updated.

[0130] B4. Play the rendered image displayed on the client through the above close-up shots.

[0131] In this embodiment, when a target operation is detected in the close-up view, the second vertex position of the close-up view is determined based on the distance change corresponding to the target operation (i.e., the close-up distance change) and the first vertex position of the close-up view. Since the first vertex position of the close-up view is the vertex position corresponding to the close-up view before the target operation is detected, and the second vertex position of the close-up view is the vertex position corresponding to the close-up view after the target operation is detected, and both vertex positions include vertex coordinates and dimensions, and the vertex coordinates are coordinates in a coordinate system established based on the viewport, the second vertex position of the close-up view can reflect the graphic range of the texture map projected onto the viewport. Therefore, the texture map portion that needs to be rendered can be determined based on the second vertex position of the close-up view, thereby enabling the image displayed by the close-up view playback client to be updated according to the detected target operation.

[0132] In some embodiments, considering that the close-up view is determined based on the view captured by the viewfinder in the panoramic view playback client, in order to ensure the consistency between the close-up view and the view captured by the viewfinder, the position of the viewfinder in the panoramic view playback client needs to be adjusted in tandem after the close-up view is adjusted. That is, after B2 above, it also includes:

[0133] The position of the viewfinder is corrected, and the position of the viewfinder includes screen coordinates, or includes screen coordinates and size.

[0134] In this embodiment, the relative position of the texture map and the close-up viewport is converted into the relative position of the panoramic viewport and the viewfinder to calculate the corrected position of the viewfinder, so that the image selected by the corrected viewfinder is consistent with the close-up image.

[0135] For example, assuming the second vertex position of the close-up view is (vertexX+Δx″,vertexY+Δy″,sideW*ΔratioW,sideH*ΔratioH), substituting the vertex coordinates (vertexX+Δx″,vertexY+Δy″) representing the lower left corner of the texture map into formula (2), we can calculate the normalized coordinates (x,y) of the upper left corner of the close-up viewport relative to the upper left corner of the texture map. Combined with... Figure 8 It can be seen that the width and height ratios of the close-up viewport relative to the texture map are (2 / sideW*ΔratioW, 2 / sideH*ΔratioH / 2), and the relative position of the texture map and the close-up viewport is consistent with the relative position of the panoramic viewport and the viewfinder. Therefore, (x,y,2 / sideW*ΔratioW,2 / sideH*ΔratioH / 2) is equivalent to the viewfinder position relative to the texture map in the panoramic viewport. The (x,y,2 / sideW*ΔratioW,2 / sideH*ΔratioH / 2) corresponds to (areaX,areaY,ratioW,ratioH) in formulas (5) and (6), respectively. That is, the calculated "x" is substituted into "areaX" in formula (5), and the calculated "y" is substituted into "areaY" in formula (5). The "(x,y,2 / sideW*ΔratioW,2 / sideH*ΔratioH / 2" in formula (5) is the same as the "(x,y,2 / sideW*ΔratioW,2 / sideH*ΔratioH / 2" in formula (5). vertex ,y vertexThe formula (5) is (vertexX+Δx″,vertexY+Δy″). Since the texture map, screen, and viewport are all determined, the other variables in formula (5) are known. Therefore, the screen coordinates of the viewfinder in the panoramic view playback client can be calculated according to formula (5). Similarly, by substituting "2 / sideW*ΔratioW" in (x,y,2 / sideW*ΔratioW,2 / sideH*ΔratioH / 2) into "ratioW" in formula (6) and "2 / sideH*ΔratioH / 2" into "ratioH" in formula (6), the length and width of the viewfinder in the panoramic view playback client can be calculated. The position of the viewfinder on the screen can be adjusted according to the calculated screen coordinates, length, and width of the viewfinder. That is, by correcting the viewfinder, the real-time linkage from the screen display range of the close-up view playback client to the viewfinder of the panoramic view playback client is realized, ensuring the consistency between the viewfinder's captured image in the panoramic view and the image displayed in the close-up view.

[0136] To more clearly describe the process of linking control of the viewfinder in the panoramic view when performing target operations on a close-up shot, the following will combine... Figure 11 Describe it.

[0137] exist Figure 11 In the context of texture mapping, the close-up viewport is used to select the graphic range on which the texture mapping is projected.

[0138] When a user performs a pan (or zoom) operation on a close-up image on the screen of the close-up playback client, the playback device calculates the change in close-up distance corresponding to the pan (or zoom) operation. Based on this change in close-up distance and the first vertex position of the close-up image (i.e., the position before the user performs the pan (or zoom) operation), it determines the position of the close-up image after the user performs the pan (or zoom) operation (i.e., the second vertex position of the close-up image). The corresponding texture map is then rendered based on the determined second vertex position of the close-up image to update the close-up image. Simultaneously, the viewfinder's position on the screen of the panoramic playback client is calculated based on the second vertex position of the close-up image (this position includes screen coordinates, or screen coordinates and dimensions), and the viewfinder is corrected based on the calculated position.

[0139] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0140] Corresponding to the dual-screen display method described in the above embodiments, Figure 12A structural block diagram of a dual-screen display device provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0141] Reference Figure 12 The dual-screen display device 12 is applied to a playback device including a panoramic view playback client. The dual-screen display device 12 includes: a panoramic distance change determination module 121, a second screen coordinate determination module 122, a vertex coordinate determination module 123, a texture mapping rendering module 124, and a screen display module 125. Wherein:

[0142] The panoramic distance change determination module 121 is used to determine the distance change corresponding to the target operation when the panoramic image displayed by the panoramic image playback client detects the target operation, and obtain the panoramic distance change. The panoramic image is obtained by rendering the projection information of the texture map projected into the viewport. The target operation includes translation operation and / or scaling operation.

[0143] The second screen coordinate determination module 122 for the panoramic image is used to determine the second screen coordinate of the panoramic image based on the panoramic distance change and the first screen coordinate of the panoramic image. The first screen coordinate of the panoramic image is the screen coordinate of the panoramic image before the target operation is detected. The screen coordinate is the coordinate under the screen coordinate system established with the screen of the panoramic image playback client. The second screen coordinate of the panoramic image is the screen coordinate of the panoramic image after the target operation is detected.

[0144] The vertex coordinate determination module 123 for panoramic view is used to determine the vertex coordinates of the panoramic view based on the second screen coordinates of the panoramic view and a preset screen vertex mapping relationship. The vertex coordinates are coordinates in the vertex coordinate system established with the viewport, and the preset screen vertex mapping relationship is used to reflect the mapping relationship between the screen coordinate system and the vertex coordinate system.

[0145] The texture mapping rendering module 124 is used to render texture maps corresponding to the vertex coordinates of the aforementioned panoramic image.

[0146] The screen display module 125 is used to display the rendered screen on the aforementioned panoramic screen playback client.

[0147] In this embodiment, by changing the vertex positions corresponding to the texture map, the range of the image displayed on the screen of the panoramic image playback client changes, thereby increasing the flexibility of the image displayed by the panoramic image playback client and increasing the probability that the image displayed by the panoramic image playback client meets the user's needs. Furthermore, since the display range of the panoramic image is adjustable, when the framing range corresponding to the viewfinder is small, the panoramic image can be magnified to synchronously magnify the viewfinder. In this way, the user can more accurately determine whether the framing range of the viewfinder is the range the user actually wants to select based on the magnified viewfinder, thus improving the accuracy of the framing range.

[0148] In some embodiments, the panoramic image playback client includes a viewfinder. The dual-view display device 12 provided in this application, when displaying the rendered image through the panoramic image playback client, further includes:

[0149] A first viewfinder correction module is used to correct the viewfinder so that the first relative position and the second relative position remain unchanged. The first relative position is the relative position of the viewfinder's framing area on the texture map and the texture map before the target operation is detected. The second relative position is the relative position of the corrected viewfinder's framing area on the texture map and the texture map.

[0150] In some embodiments, the viewfinder first correction module is specifically used for:

[0151] The position of the viewfinder is corrected, and the position of the viewfinder includes screen coordinates, or includes screen coordinates and size.

[0152] In some embodiments, the playback device further includes a close-up view playback client, which is used to magnify and display the panoramic view captured by the viewfinder. The dual-view display device 12 further includes:

[0153] The viewfinder distance change determination module is used to determine the distance change corresponding to the target operation when the viewfinder detects the target operation, and obtain the viewfinder distance change.

[0154] The viewfinder update module is used to update the viewfinder based on the change in viewfinder distance and the first position of the viewfinder, wherein the first position of the viewfinder is the position of the viewfinder before the target operation is detected, and the position includes the coordinate information and / or size of the viewfinder.

[0155] The close-up view update module is used to update the view displayed on the close-up view playback client according to the updated position of the viewfinder.

[0156] In some embodiments, the first position of the viewfinder is the position of the viewfinder in the texture coordinate system before the target operation is detected, and the viewfinder update module is specifically used for:

[0157] The second position of the viewfinder is determined based on the change in viewfinder distance and the first position of the viewfinder. The second position of the viewfinder is the position of the viewfinder in the texture coordinate system after the target operation is detected.

[0158] The viewfinder is updated based on its second position.

[0159] In some embodiments, the dual-screen display device 12 further includes:

[0160] The close-up distance change determination module is used to determine the distance change corresponding to the target operation when the close-up screen displayed by the client is detected to perform the aforementioned close-up screen operation, and to obtain the close-up distance change.

[0161] The second vertex position determination module of the close-up image is used to determine the second vertex position of the close-up image based on the change in the close-up distance and the first vertex position of the close-up image. The first vertex position of the close-up image is the vertex position of the close-up image before the target operation is detected, and the second vertex position of the close-up image is the vertex position of the close-up image after the target operation is detected. Both the first vertex position and the second vertex position of the close-up image include vertex coordinates and dimensions.

[0162] The close-up rendering module is used to render the texture map corresponding to the second vertex position of the aforementioned close-up image.

[0163] The close-up display module is used to display the rendered image on the client side through the aforementioned close-up display.

[0164] In some embodiments, the dual-screen display device 12 provided in this application further includes:

[0165] The viewfinder second correction module is used to correct the position of the viewfinder after determining the second vertex position of the close-up image based on the change in the close-up distance and the first vertex position of the close-up image. The position of the viewfinder includes screen coordinates, or screen coordinates and size.

[0166] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0167] Figure 13 This is a schematic diagram of the structure of a playback device provided in one embodiment of this application. Figure 13 As shown, the playback device 13 of this embodiment includes: at least one processor 130 ( Figure 13 The diagram shows only one processor, a memory 131, a computer program 132 stored in the memory 131 and executable on the at least one processor 130, and a panoramic view playback client 133 and a close-up view playback client 134 that display the image processed by the processor 130. When the processor 130 executes the computer program 132, it combines the panoramic view playback client 133 and the close-up view playback client 134 to implement the steps in any of the above method embodiments.

[0168] The playback device 13 may include, but is not limited to, a processor 130 and a memory 131. Those skilled in the art will understand that... Figure 13 This is merely an example of playback device 13 and does not constitute a limitation on playback device 13. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0169] The processor 130 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0170] In some embodiments, the memory 131 may be an internal storage unit of the playback device 13, such as a hard disk or memory of the playback device 13. In other embodiments, the memory 131 may be an external storage device of the playback device 13, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the playback device 13. Further, the memory 131 may include both internal and external storage units of the playback device 13. The memory 131 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 131 can also be used to temporarily store data that has been output or will be output.

[0171] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0172] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0173] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0174] This application provides a computer program product that, when run on a playback device, enables the playback device to implement the steps described in the various method embodiments above.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing / playback device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0176] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0177] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0179] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A dual-screen display method, characterized in that, The dual-screen display method, applied to playback devices including panoramic image playback clients, includes: When a target operation is detected in the panoramic image displayed on the panoramic image playback client, the distance change corresponding to the target operation is determined to obtain the panoramic distance change. The panoramic image is obtained by rendering the projection information of the texture map projected into the viewport. The target operation includes translation operation and / or scaling operation. The second screen coordinates of the panoramic image are determined based on the change in panoramic distance and the first screen coordinates of the panoramic image. The first screen coordinates of the panoramic image are the screen coordinates of the panoramic image before the target operation is detected. The screen coordinates are the coordinates in the screen coordinate system established by the screen of the panoramic image playback client. The second screen coordinates of the panoramic image are the screen coordinates of the panoramic image after the target operation is detected. The vertex coordinates of the panoramic image are determined based on the second screen coordinates of the panoramic image and a preset screen vertex mapping relationship. The vertex coordinates are coordinates in the vertex coordinate system established with the viewport, and the preset screen vertex mapping relationship is used to reflect the mapping relationship between the screen coordinate system and the vertex coordinate system. Render the texture map corresponding to the vertex coordinates of the panoramic image; The panoramic view is displayed and rendered by the client.

2. The dual-screen display method as described in claim 1, characterized in that, The panoramic image playback client includes a viewfinder, and when displaying the rendered image through the panoramic image playback client, it further includes: The viewfinder is corrected so that the first relative position and the second relative position remain unchanged. The first relative position is the relative position of the viewfinder's framing area on the texture map and the texture map before the target operation is detected. The second relative position is the corrected relative position of the viewfinder's framing area on the texture map and the texture map.

3. The dual-screen display method as described in claim 2, characterized in that, The correction of the viewfinder includes: The position of the viewfinder is corrected, and the position of the viewfinder includes screen coordinates, or includes the screen coordinates and size.

4. The dual-screen display method as described in claim 2 or 3, characterized in that, The playback device further includes a close-up view playback client, which is used to magnify and display the panoramic view captured by the viewfinder. The dual-view display method further includes: When the viewfinder detects a target operation, the distance change corresponding to the target operation is determined to obtain the viewfinder distance change. The viewfinder is updated based on the change in viewfinder distance and the first position of the viewfinder, wherein the first position of the viewfinder is the position of the viewfinder before the target operation is detected, and the position includes the coordinate information and / or size of the viewfinder; The viewfinder position is updated to update the image displayed on the close-up playback client.

5. The dual-screen display method as described in claim 4, characterized in that, The first position of the viewfinder is the position of the viewfinder in the texture coordinate system before the target operation is detected. Updating the viewfinder based on the viewfinder distance change and the first position includes: The second position of the viewfinder is determined based on the change in viewfinder distance and the first position of the viewfinder. The second position of the viewfinder is the position of the viewfinder in the texture coordinate system after the target operation is detected. The viewfinder is updated based on its second position.

6. The dual-screen display method as described in claim 4, characterized in that, The dual-screen display method further includes: If a target operation is detected in the close-up image displayed on the client side of the close-up image playback, the distance change corresponding to the target operation is determined, and the close-up distance change is obtained. The second vertex position of the close-up image is determined based on the change in the close-up distance and the first vertex position of the close-up image. The first vertex position of the close-up image is the vertex position of the close-up image before the target operation is detected, and the second vertex position of the close-up image is the vertex position of the close-up image after the target operation is detected. Both the first vertex position and the second vertex position of the close-up image include vertex coordinates and dimensions. Render the texture map corresponding to the second vertex position of the close-up image; The client displays the rendered image through the close-up shot.

7. The dual-screen display method as described in claim 6, characterized in that, After determining the second vertex position of the close-up image based on the change in close-up distance and the first vertex position of the close-up image, the method further includes: The position of the viewfinder is corrected, and the position of the viewfinder includes screen coordinates, or includes the screen coordinates and size.

8. A dual-screen display device, characterized in that, Playback devices applicable to panoramic image playback clients include: A panoramic distance change determination module is used to determine the distance change corresponding to the target operation when the panoramic image displayed by the panoramic image playback client detects the target operation, and obtain the panoramic distance change. The panoramic image is obtained by rendering the projection information of the texture map projected into the viewport. The target operation includes translation operation and / or scaling operation. The second screen coordinate determination module for the panoramic image is used to determine the second screen coordinate of the panoramic image based on the panoramic distance change and the first screen coordinate of the panoramic image. The first screen coordinate of the panoramic image is the screen coordinate of the panoramic image before the target operation is detected. The screen coordinate is the coordinate under the screen coordinate system established with the screen of the panoramic image playback client. The second screen coordinate of the panoramic image is the screen coordinate of the panoramic image after the target operation is detected. A vertex coordinate determination module for panoramic images is used to determine the vertex coordinates of the panoramic images based on the second screen coordinates of the panoramic images and a preset screen vertex mapping relationship. The vertex coordinates are coordinates in a vertex coordinate system established with the viewport, and the preset screen vertex mapping relationship is used to reflect the mapping relationship between the screen coordinate system and the vertex coordinate system. The texture mapping rendering module is used to render texture maps corresponding to the vertex coordinates of the panoramic image; The screen display module is used to display the rendered screen through the panoramic screen playback client.

9. A playback device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images