Immersive video super-resolution method and related equipment

Abstract

The invention provides an immersive video super-resolution method, and the method comprises the steps: decoding a video file, distributing an independent key frame propagation branch for a key frame, obtaining the features of the key frame, and guaranteeing that the information recorded in the key frame can be fully utilized; a forward propagation branch and a backward propagation branch are allocated to the key frame, so that the non-key frame can further utilize information recorded by the key frame; the forward propagation branches and the backward propagation branches corresponding to the non-key frames are alternately arranged, so that unnecessary propagation branches are reduced, and the calculation amount required by the super-resolution of the panoramic video is reduced; after the corresponding features of the key frame and the non-key frame are obtained, super-resolution is carried out on the key frame and the non-key frame, and a high-definition key frame, a first high-definition non-key frame and a second high-definition non-key frame are obtained and spliced into a high-definition immersive video file.

Description

Technical Field

[0001] The present application relates to the technical field of video super-resolution, and in particular to an immersive video super-resolution method and related equipment. Background Art

[0002] With the development of technology and social innovation, immersive video technology is becoming more and more popular among the public. Unlike traditional videos that capture scenes from a single perspective, immersive videos achieve full coverage of scenes by shooting from multiple angles in situ. Viewers can use a dedicated head-mounted display to control the perspective through head movements, or use a mouse and keyboard to explore the video on a computer screen, breaking the limitations of traditional video perspectives and providing an immersive viewing experience. Compared with traditional videos, in order to provide a satisfactory immersive experience, panoramic videos usually need to be transmitted at 4K or 8K resolution, which not only increases the amount of data, but also places higher requirements on network bandwidth.

[0003] In the transmission of traditional videos, in order to reduce the bandwidth burden of servers and enable a wider user group to access immersive videos, low-quality videos are currently transmitted over the Internet, and super-resolution of low-quality videos is performed on the client to achieve transmission of traditional videos.

[0004] The current super-resolution model for traditional videos has a relatively complex structure. Directly applying it to the super-resolution operation of immersive videos will significantly increase the amount of data that the client device needs to process, resulting in too long a super-resolution processing time for immersive videos on client devices with poor performance. Summary of the invention

[0005] In view of this, the purpose of this application is to propose an immersive video super-resolution method and related equipment.

[0006] Based on the above purpose, the present application provides an immersive video super-resolution method, including: decoding a video file, determining key frames and non-key frames. Key frame propagation branches, forward propagation branches and backward propagation branches are assigned to key frames. Among them, the key frame propagation branch is configured to calculate the key frame features of the current key frame based on the previous key frame, the key frame features of the previous key frame and the current key frame. The forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch and the current frame. The backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch and the current frame. Forward propagation branches and backward propagation branches are assigned to non-key frames, and only one of the forward propagation branches and the backward propagation branches is assigned to each non-key frame, and the propagation branches assigned to two adjacent non-key frames are different. Based on the key frame features, the first features and the second features corresponding to the key frames, super-resolution is performed on the key frames to obtain high-definition key frames. Based on the first feature corresponding to the non-key frame, the non-key frame is super-resolved to obtain a first high-definition non-key frame. Based on the second feature corresponding to the non-key frame, the non-key frame is super-resolved to obtain a second high-definition non-key frame. Based on the high-definition key frame, the first high-definition non-key frame and the second high-definition non-key frame, a high-definition immersive video file is obtained.

[0007] In some embodiments, the key frame propagation branch includes a key frame optical flow estimation module, a key frame feature alignment module and a key frame residual module. The key frame propagation branch is specifically configured as follows: the key frame and the previous key frame are input into the key frame optical flow estimation module, the key frame optical flow estimation module compares the feature difference between the key frame and the previous key frame, and outputs the key frame difference information. The key frame difference information and the key frame feature of the previous key frame are input into the key frame feature alignment module, the key frame feature alignment module distorts the previous key frame feature to align the previous key frame feature with the key frame difference information, and outputs the key frame alignment information. The key frame and the key frame alignment information are input into the key frame residual module, and the key frame residual module outputs the key frame feature.

[0008] In some embodiments, the forward propagation branch includes a first optical flow estimation module, a first feature alignment module and a first residual module. The forward propagation branch is specifically configured as follows: the current frame and the previous frame including the forward propagation branch are input into the first optical flow estimation module, and the first optical flow estimation module compares the feature difference between the current frame and the previous frame including the forward propagation branch, and outputs the first difference information. The first difference information and the first feature output by the previous forward propagation branch are input into the first feature alignment module, and the first feature alignment module distorts the first feature output by the previous forward propagation branch so that the first feature output by the previous forward propagation branch is aligned with the first difference information, and outputs the first alignment information. The current frame and the first alignment information are input into the first residual module, and the first residual module outputs the first feature.

[0009] In some embodiments, the backward propagation branch includes a second optical flow estimation module, a second feature alignment module and a second residual module. The backward propagation branch is specifically configured as follows: the current frame and the next frame including the backward propagation branch are input into the second optical flow estimation module, and the second optical flow estimation module compares the feature difference between the current frame and the next frame including the backward propagation branch, and outputs the second difference information. The second difference information and the second feature output by the next backward propagation branch are input into the second feature alignment module, and the second feature alignment module distorts the second feature output by the next backward propagation branch to align the next second feature with the second difference information, and outputs the second alignment information. The current frame and the second alignment information are input into the second residual module, and the second residual module outputs the second feature.

[0010] In some embodiments, super-resolution is performed on the key frame based on the key frame features, the first feature and the second feature corresponding to the key frame, specifically including: rearranging the key frame features, the first feature and the second feature, and super-resolution is performed on the key frame based on the sub-pixel convolution method.

[0011] In some embodiments, performing super-resolution on the key frame based on the sub-pixel convolution method further includes: calculating the difference between each point in the key frame high-definition image and the key frame based on the Charbonnier loss function as a loss value.

[0012] In some embodiments, after decoding the video file and determining the key frame and the non-key frame, the method further includes: segmenting the key frame to obtain a segmented image. The segmented image close to the first edge of the key frame is spliced ​​to the second edge of the key frame. The first edge and the second edge can be spliced ​​into a continuous image.

[0013] In some embodiments, it includes: a decoding module for decoding a video file and determining key frames and non-key frames. A propagation branch allocation module for allocating key frame propagation branches, forward propagation branches and backward propagation branches to key frames. Among them, the key frame propagation branch is configured to calculate the key frame features of the current key frame based on the previous key frame, the key frame features of the previous key frame and the current key frame. The forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch and the current frame. The backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch and the current frame. And forward propagation branches and backward propagation branches are allocated to non-key frames, and only one of the forward propagation branches and the backward propagation branches is allocated to each non-key frame, and the propagation branches allocated to two adjacent non-key frames are different. A key frame super-resolution module is used to perform super-resolution on the key frame based on the key frame features, the first feature and the second feature corresponding to the key frame to obtain a high-definition key frame. The first super-resolution module is used to perform super-resolution on the non-key frame based on the first feature corresponding to the non-key frame to obtain a first high-definition non-key frame. The second super-resolution module is used to perform super-resolution on the non-key frame based on the second feature corresponding to the non-key frame to obtain a second high-definition non-key frame. The video generation module is used to obtain a high-definition immersive video file based on the high-definition key frame, the first high-definition non-key frame and the second high-definition non-key frame.

[0014] In some embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the program.

[0015] In some embodiments, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above methods.

[0016] It can be seen from the above embodiments that the immersive video super-resolution method provided by the present application decodes the video file and assigns independent key frame propagation branches to key frames to obtain key frame features, thereby ensuring that the information recorded in the key frames can be fully utilized; and assigns forward propagation branches and backward propagation branches to key frames, so that non-key frames can further utilize the information recorded in the key frames; by alternating the forward propagation branches and backward propagation branches corresponding to non-key frames, unnecessary propagation branches are reduced, thereby reducing the amount of computation required for panoramic video super-resolution; after obtaining the features corresponding to the key frames and non-key frames, super-resolution is performed on the key frames and non-key frames respectively to obtain high-definition key frames, first high-definition non-key frames and second high-definition non-key frames, and they are spliced ​​into high-definition immersive video files. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A flowchart of an immersive video super-resolution method provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a propagation branch allocation method provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of the structure of the propagation branch provided in the embodiment of the present application;

[0021] Figure 4 A schematic structural diagram of an immersive video super-resolution device provided by another embodiment of the present application;

[0022] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0024] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0025] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0026] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0027] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0028] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0029] Immersive video, usually referring to spherical video or panoramic video, is a cutting-edge multimedia expression technique. Unlike traditional videos that capture scenes from a single perspective, panoramic videos achieve full coverage of scenes by shooting from multiple angles in situ. Viewers can use a dedicated head-mounted display to control the perspective through head movements, or use a mouse and keyboard to explore the video on a computer screen, breaking the limitations of traditional video perspectives and providing an immersive viewing experience. With technological development and social innovation, panoramic video technology is becoming more and more popular with the public and is being used in many industries such as education, medical care, tourism, retail, and entertainment. For example, in the tourism industry, panoramic videos allow users to experience scenery from all over the world at home; in the entertainment industry, panoramic videos combined with dynamic seats can simulate experiences such as roller coasters, giving users an immersive feeling. The maturity of technology and the reduction in costs have promoted the popularization of panoramic videos, and users can watch panoramic content through various devices, including professional head-mounted devices, computer screens, and even mobile phone screens. However, the high data volume of panoramic videos poses a challenge to network transmission. Compared with traditional videos, in order to provide a satisfactory immersive experience, panoramic videos usually need to be transmitted in 4K or 8K resolution, which not only increases the amount of data, but also places higher requirements on network bandwidth. For example, 8K video streaming may require a bandwidth of at least 100Mbps.

[0030] In the field of computer vision, super-resolution technology has attracted much attention in recent years. This technology uses algorithms to upsample low-resolution images or videos to high resolution, and is widely used in video restoration, deblurring, denoising and other fields, and has gradually expanded to video transmission. By performing super-resolution processing on low-resolution videos on the client side instead of directly transmitting high-resolution videos from the server, the bandwidth burden on the server side can be significantly reduced, allowing users to enjoy high-definition videos with limited bandwidth.

[0031] As society's requirements for panoramic video quality of experience increase, super-resolution technology is particularly critical in helping user groups with limited resources. The value of studying panoramic video super-resolution technology is that it can effectively reduce the amount of data required for panoramic video transmission, so that users can enjoy high-quality, high-resolution panoramic videos even in ordinary network environments by receiving lower-quality video data and performing super-resolution processing locally, thereby enhancing immersion and realism. However, it is worth noting that the time required for video super-resolution processing is also an important factor affecting the user's viewing experience. Too long processing time may cause users to give up watching.

[0032] In view of this, the purpose of this study is to develop a more efficient deep learning-driven panoramic video super-resolution algorithm by optimizing the video super-resolution network architecture. The goal of this algorithm is to lower the threshold for users to enjoy immersive content, promote the continued development of the panoramic video field, and make the immersive experience accessible to a wider user group.

[0033] At present, the commonly used super-resolution models often have complex structures, which makes them have certain limitations in practical deployment and application. Video super-resolution technology essentially involves the processing of huge data sets, and panoramic videos have high requirements for resolution, which significantly increases the amount of data required to be processed compared to traditional video formats, thus significantly increasing the complexity of the task. To address this problem, this paper aims to simplify the traditional video super-resolution model and propose a super-resolution method for panoramic videos that can accelerate the inference process while ensuring reasonable quality.

[0034] In addition, current video super-resolution algorithms are mainly targeted at planar videos, and there is little research on panoramic videos. However, due to the particularity of panoramic videos, in order to better utilize the information in the panoramic video frames, it is necessary to preprocess the panoramic video before inputting it into the model.

[0035] In view of the complexity of the information propagation mechanism of existing video super-resolution models, the present invention simplifies the structure and optimizes the processing flow. Specifically, this paper preprocesses the panoramic video to make it more efficient in information utilization. In addition, this paper proposes a simplified two-way propagation strategy, which has fewer propagation branches and reduces the amount of calculation compared to the original two-way propagation strategy, aiming to improve the efficiency of super-resolution training and reasoning. At the same time, key frames are very important frames in video encoding and contain the most complete information. This paper combines the method of super-resolution of key frames to ensure that the quality of the video remains as unchanged as possible. Based on this, this study designed a panoramic video super-resolution method based on deep learning, using incomplete two-way propagation and super-resolution key frames to speed up super-resolution reasoning while ensuring the quality of super-resolution.

[0036] like Figure 1 As shown, the present application provides an immersive video super-resolution method, comprising:

[0037] Step S1, decoding the video file and determining key frames and non-key frames.

[0038] In this embodiment, the video file is a panoramic video, which is encoded into a video file by expanding a 360° panoramic video into a flat video. When playing the video file, the panoramic video is decoded by the player and the flat video is mapped into the panoramic space, thereby realizing the playback of the panoramic video. The key frame includes complete video information, and the non-key frame is a frame similar to the key frame image during the video compression process. By deleting some repeated information, the amount of information in the video is reduced, and the complete image of the current non-key frame can be restored by referring to the key frame or other non-key frames.

[0039] Before inputting the panoramic video into the super-resolution network, the video needs to be preprocessed first. This is because the super-resolution neural network generally receives input in the format of a sequence of pictures, and the encoded video does not exist in the form of pictures. In the video encoding stream, there are three types of frames: I frames (key frames), P frames (predicted frames), and B frames (bidirectionally predicted frames) used to compress video data. I frames are independently encoded and contain complete image data, so the file size is large, but the video can be played from any I frame. P frames rely on the previous I frame or P frame, only store difference information, and reconstruct the current frame by referring to the previous frame, thereby improving compression efficiency. B frames are encoded with reference to the information of the previous and next frames, and achieve higher compression efficiency by storing the difference information between frames, but the decoding process is more complicated. Obviously, I frames (key frames) contain more complete and reliable information than other P frames and B (non-key frames), and the decoder can easily obtain the position information of the key frames when decoding, and the key frames can be marked according to the information of the decoder when outputting.

[0040] Step S2, assigning a key frame propagation branch, a forward propagation branch and a backward propagation branch to the key frame. The key frame propagation branch is configured to calculate the key frame feature of the current key frame based on the previous key frame, the key frame feature of the previous key frame and the current key frame. The forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch and the current frame. The backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch and the current frame.

[0041] Among them, in video super-resolution processing, propagation refers to the process of transmitting and sharing information between multiple video frames. By utilizing the information of the front and back frames, motion and details can be better captured and the quality of the current frame can be enhanced. Three propagation branches are used in this application, namely forward propagation branch, backward propagation branch and key frame propagation branch. The structures of these propagation branches are the same, taking the feature information of the current frame, the previous frame (reference frame) and the previous propagation branch as input, outputting the feature information obtained by the current propagation branch, and passing it to the aggregation & upsampling module of the next propagation branch and the current frame. The forward propagation branch transmits the feature information forward along the time sequence, the backward propagation branch transmits the feature information in reverse along the time sequence, and the key frame propagation branch transmits the key frame information forward along the time sequence.

[0042] Step S3, allocating a forward propagation branch and a backward propagation branch to the non-key frames, and only allocating one of the forward propagation branch and the backward propagation branch to each non-key frame, and the propagation branches allocated to two adjacent non-key frames are different.

[0043] like Figure 2 As shown, since the calculation of the propagation branch is relatively complicated, if two forward and backward propagation links are designed for each frame, a large number of convolution operations need to be performed. These operations consume a lot of time and require high computing performance, which is not conducive to the experience of immersive content by users with low hardware performance. Therefore, in this application, by assigning three propagation branches, namely key frame propagation branches, forward propagation branches and backward propagation branches to key frames, and setting forward propagation branches and backward propagation branches for non-key frame intervals, the amount of calculation required for video super-resolution can be reduced while ensuring the quality of video super-resolution. And by assigning key frame propagation branches independent of forward propagation branches and backward propagation branches to key frames, the complete information recorded in the key frames can be better utilized, thereby improving the quality of video super-resolution.

[0044] On this basis, by connecting the key frame propagation branch, the forward propagation branch and the backward propagation branch, the three propagation branches corresponding to all frames are traversed according to the arrangement order of key frames and non-key frames, so that the three propagation branches respectively output the frame features of the corresponding frames, and based on the frame features corresponding to each frame, the current frame is super-resolution to obtain a high-definition image.

[0045] Step S4, performing super-resolution on the key frame based on the key frame feature, the first feature and the second feature corresponding to the key frame to obtain a high-definition key frame.

[0046] Step S5: Based on the first feature corresponding to the non-key frame, super-resolution is performed on the non-key frame to obtain a first high-definition non-key frame.

[0047] Step S6: Based on the second feature corresponding to the non-key frame, super-resolution is performed on the non-key frame to obtain a second high-definition non-key frame.

[0048] Step S7, obtaining a high-definition immersive video file based on the high-definition key frame, the first high-definition non-key frame and the second high-definition non-key frame.

[0049] Among them, the high-definition key frame, the first high-definition non-key frame and the second high-definition non-key frame are encoded according to the arrangement of the original key frame and the non-key frame, so as to obtain a high-definition immersive video. Based on the above steps of this method, the low-definition immersive video can be transmitted to the terminal device, and the low-definition immersive video can be super-resolved by this method, so as to achieve the purpose of saving video transmission bandwidth, and the panoramic video file with low original file definition can also be super-resolved to obtain a clearer immersive video picture.

[0050] In some embodiments, the key frame propagation branch includes a key frame optical flow estimation module, a key frame feature alignment module and a key frame residual module.

[0051] In this embodiment, if Figure 3 As shown, the key frame optical flow estimation module, the key frame feature alignment module and the key frame residual module are connected in sequence, taking the current key frame, the previous key frame and the key frame features of the previous key frame as input, and outputting the key frame features of the current key frame after being processed by the key frame optical flow estimation module, the key frame feature alignment module and the key frame residual module.

[0052] The key frame propagation branch is specifically configured as follows: the key frame and the previous key frame are input into the key frame optical flow estimation module, the key frame optical flow estimation module compares the feature difference between the key frame and the previous key frame, and outputs the key frame difference information.

[0053] Among them, due to the motion and deformation in the video, the image content between the frames analyzed by the propagation branch may be displaced and distorted. In this application, each propagation branch has the same structure as the traditional propagation branch, such as Figure 3 As shown, x i Represents the current frame of the input. The current frame will be the previous frame x in the current propagation direction. i-1 They are input together into the key frame optical flow estimation module, which is responsible for analyzing the differences in features between the two frames.

[0054] The key frame difference information and the key frame features of the previous key frame are input into the key frame feature alignment module, and the key frame feature alignment module distorts the previous key frame features to align the previous key frame features with the key frame difference information, and outputs the key frame alignment information.

[0055] Among them, the key frame difference information calculated by the key frame optical flow estimation module will be combined with the key frame feature f from the previous key frame propagation branch i-1 They are input together into the key frame feature alignment module, which uses the key frame difference information to distort the features from the previous frame to achieve the effect of alignment with the current frame, and preferably outputs the key frame alignment information.

[0056] The keyframe and keyframe alignment information are input into the keyframe residual module, and the keyframe residual module outputs the keyframe features.

[0057] Among them, by inputting the key frame alignment information and the current frame into the residual module together, the output key frame features will be used for subsequent propagation and aggregation and upsampling of the current frame.

[0058] In some embodiments, Figure 3 As shown, the forward propagation branch includes a first optical flow estimation module, a first feature alignment module and a first residual module.

[0059] The forward propagation branch is specifically configured as follows: the current frame and the previous frame including the forward propagation branch are input into the first optical flow estimation module, the first optical flow estimation module compares the feature difference between the current frame and the previous frame including the forward propagation branch, and outputs first difference information.

[0060] The first difference information and the first feature output by the previous forward propagation branch are input into a first feature alignment module. The first feature alignment module distorts the first feature output by the previous forward propagation branch so as to align the first feature output by the previous forward propagation branch with the first difference information, and outputs the first alignment information.

[0061] The current frame and the first alignment information are input into a first residual module, and the first residual module outputs a first feature.

[0062] In this embodiment, similar to the key frame propagation branch, the forward propagation branch takes the current frame, the previous frame and the first feature of the previous frame as input, and is processed by the first optical flow estimation module, the first feature alignment module and the first residual module to output the first feature of the current frame.

[0063] It should be noted that since the forward propagation branch can be assigned to key frames for outputting the first feature of key frames, and can also be assigned to non-key frames for outputting the first feature of non-key frames, the types of the previous frame and the current frame need to be determined according to the arrangement order of the key frames and non-key frames, and the processing methods of different types of previous frames and current frames can be the same or different.

[0064] In some embodiments, Figure 3As shown, the backward propagation branch includes a second optical flow estimation module, a second feature alignment module and a second residual module.

[0065] The backward propagation branch is specifically configured as follows: the current frame and the next frame including the backward propagation branch are input into the second optical flow estimation module, the second optical flow estimation module compares the feature difference between the current frame and the next frame including the backward propagation branch, and outputs second difference information.

[0066] The second difference information and the second feature output by the next backward propagation branch are input into the second feature alignment module. The second feature alignment module distorts the second feature output by the next backward propagation branch to align the next second feature with the second difference information, and outputs the second alignment information.

[0067] The current frame and the second alignment information are input into a second residual module, and the second residual module outputs a second feature.

[0068] In this embodiment, similar to the forward propagation branch, the backward propagation branch obtains the second feature of the current frame based on the current frame, the next frame and the second feature of the next frame, and both the next frame and the current frame can be key frames or non-key frames.

[0069] In some embodiments, step S4 specifically includes:

[0070] Step S41, rearrange the key frame features, the first features and the second features, and perform super-resolution on the key frame based on a sub-pixel convolution method.

[0071] In this embodiment, after obtaining the key frame features, the first features and the second features, the key frame can be super-resolutioned by using a sub-pixel convolution method, namely, a Pixel-Shuffle method. Specifically, the r generated by the key frame needs to be 2 The feature maps of the size of H×W×C channels are rearranged to generate a high-definition image of size rH×rW×C. Where r is the upsampling magnification, H and W are the height and width of the image respectively, and C is the color channel. For example, an RGB image has 3 color channels. The upsampling process can be formally described as follows,

[0072]

[0073] PS refers to the Pixel-Shuffle operation. SR and I LR Respectively represent the generated high-definition image and the original low-definition image. = is used to represent I SR That is how the image restored using super resolution is generated. From right to left, PS (I LR ) refers to the PS operation on low-definition pictures. refers to the r generated by the key frame 2 channels (which can be understood as different 2 The feature map of size H×W×C is extracted by feature extraction. rH×rW×C It refers to a high-definition image of size rH×rW×C obtained by rearranging the feature map.

[0074] Preferably, step S5 may further include:

[0075] Step S51, rearrange the first features, perform super-resolution on the non-key frame based on a sub-pixel convolution method, and obtain a first high-definition non-key frame.

[0076] Preferably, step S6 may further include:

[0077] Step S61, rearrange the second features, and perform super-resolution on non-key frames based on a sub-pixel convolution method.

[0078] The super-resolution operation on non-key frames based on the Pixel-Shuffle method is the same as the super-resolution operation process on key frames described above, and will not be described in detail here.

[0079] In some embodiments, step S41 further includes:

[0080] Step S411 calculates the difference between each point in the key frame high-definition image and the key frame based on the Charbonnier loss function as a loss value.

[0081] In this embodiment, when converting a panoramic video into a flat video, in order to facilitate the storage of the video file, the panoramic video is usually cut along the longitude of the panoramic video, and the panoramic video is stretched and deformed, and converted into a rectangular video for storage. Therefore, the closer the area is to the two poles of the panoramic video, the more serious the stretching and deformation is, and the reliability of the residual calculation result of this part is low. In order to avoid the influence of the stretched part and cause inaccurate residual calculation, it is necessary to assign weights to each point in the panoramic video image. The definition of the weight of a certain point (i, j) is expressed as follows:

[0082]

[0083] The loss value is calculated based on the Charbonnier loss function. The calculation formula is as follows:

[0084]

[0085] Among them, M and N represent the width and height of the image respectively. Represents a point on the predicted image, I i,j Represents a point on the real image.

[0086] In some embodiments, after step S1, the method further includes:

[0087] Step S11, segment the key frame to obtain a segmented image.

[0088] Step S12: splicing the segmented images close to the first edge of the key frame to the second edge of the key frame.

[0089] The first edge and the second edge can be spliced ​​into a continuous image.

[0090] In this embodiment, due to the relationship between panoramic video mapping, the subject in the panoramic video screen may be located at the edge, resulting in the inability to effectively utilize continuous information, so the image needs to be properly spliced. Specifically, the panoramic video screen can be divided into several equal parts, the split image on one side of the screen is copied and spliced ​​to the other side, and the spliced ​​image is super-resolution. Optionally, in order to improve the super-resolution effect, the screen can also be divided into a grid form, the screen of at least one edge of the screen is copied and all are spliced ​​to another edge. When the screen is too complex, the number of spliced ​​edges can be increased as appropriate, and one corner of the screen can be spliced ​​to the corresponding corner. The splicing method can be reasonably modified according to actual application needs.

[0091] It can be seen from the above embodiments of the present application that the present application decodes the video file and assigns independent key frame propagation branches to the key frames to obtain key frame features, thereby ensuring that the information recorded in the key frames can be fully utilized; and assigns forward propagation branches and backward propagation branches to the key frames, so that non-key frames can further utilize the information recorded in the key frames; by alternating the forward propagation branches and backward propagation branches corresponding to the non-key frames, unnecessary propagation branches are reduced, thereby reducing the amount of computation required for panoramic video super-resolution; after obtaining the features corresponding to the key frames and non-key frames, super-resolution is performed on the key frames and non-key frames respectively to obtain high-definition key frames, first high-definition non-key frames and second high-definition non-key frames, and they are spliced ​​into high-definition immersive video files.

[0092] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0093] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0094] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides an immersive video super-resolution device.

[0095] refer to Figure 4 , the immersive video super-resolution device comprises:

[0096] The decoding module is used to decode the video file and determine the key frames and non-key frames.

[0097] The propagation branch allocation module is used to allocate key frame propagation branches, forward propagation branches and backward propagation branches to key frames. The key frame propagation branch is configured to calculate the key frame features of the current key frame based on the previous key frame, the key frame features of the previous key frame and the current key frame. The forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch and the current frame. The backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch and the current frame.

[0098] A forward propagation branch and a backward propagation branch are allocated to the non-key frames, and only one of the forward propagation branch and the backward propagation branch is allocated to each non-key frame, and the propagation branches allocated to two adjacent non-key frames are different.

[0099] The key frame super-resolution module is used to perform super-resolution on the key frame based on the key frame feature, the first feature and the second feature corresponding to the key frame to obtain a high-definition key frame.

[0100] The first super-resolution module is used to perform super-resolution on the non-key frame based on a first feature corresponding to the non-key frame to obtain a first high-definition non-key frame.

[0101] The second super-resolution module is used to perform super-resolution on the non-key frame based on a second feature corresponding to the non-key frame to obtain a second high-definition non-key frame.

[0102] The video generation module is used to obtain a high-definition immersive video file based on the high-definition key frame, the first high-definition non-key frame and the second high-definition non-key frame.

[0103] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0104] The device of the above embodiment is used to implement the corresponding immersive video super-resolution method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0105] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an immersive video super-resolution method described in any of the above embodiments is implemented.

[0106] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0107] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0108] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0109] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0110] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0111] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0112] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0113] The electronic device of the above embodiment is used to implement a corresponding immersive video super-resolution method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0114] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute an immersive video super-resolution method as described in any of the above embodiments.

[0115] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0116] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute an immersive video super-resolution method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0117] It should be noted that the embodiments of the present application can be further described in the following manner:

[0118] An immersive video super-resolution method, comprising:

[0119] Decode the video file and identify key frames and non-key frames.

[0120] A key frame propagation branch, a forward propagation branch, and a backward propagation branch are assigned to the key frame. The key frame propagation branch is configured to calculate the key frame feature of the current key frame based on the previous key frame, the key frame feature of the previous key frame, and the current key frame. The forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch, and the current frame. The backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch, and the current frame.

[0121] A forward propagation branch and a backward propagation branch are allocated to non-key frames, and only one of the forward propagation branch and the backward propagation branch is allocated to each non-key frame, and the propagation branches allocated to two adjacent non-key frames are different.

[0122] Based on the key frame features, the first features, and the second features corresponding to the key frames, super-resolution is performed on the key frames to obtain high-definition key frames.

[0123] Based on the first feature corresponding to the non-key frame, super-resolution is performed on the non-key frame to obtain a first high-definition non-key frame.

[0124] Based on the second feature corresponding to the non-key frame, super-resolution is performed on the non-key frame to obtain a second high-definition non-key frame.

[0125] A high-definition immersive video file is obtained based on the high-definition key frame, the first high-definition non-key frame and the second high-definition non-key frame.

[0126] Optionally, the key frame propagation branch includes a key frame optical flow estimation module, a key frame feature alignment module and a key frame residual module.

[0127] The key frame propagation branch is specifically configured as follows: the key frame and the previous key frame are input into the key frame optical flow estimation module, the key frame optical flow estimation module compares the feature difference between the key frame and the previous key frame, and outputs the key frame difference information.

[0128] The key frame difference information and the key frame features of the previous key frame are input into the key frame feature alignment module, and the key frame feature alignment module distorts the previous key frame features to align the previous key frame features with the key frame difference information, and outputs the key frame alignment information.

[0129] The keyframe and keyframe alignment information are input into the keyframe residual module, and the keyframe residual module outputs the keyframe features.

[0130] Optionally, the forward propagation branch includes a first optical flow estimation module, a first feature alignment module and a first residual module.

[0131] The forward propagation branch is specifically configured as follows: the current frame and the previous frame including the forward propagation branch are input into the first optical flow estimation module, the first optical flow estimation module compares the feature difference between the current frame and the previous frame including the forward propagation branch, and outputs first difference information.

[0132] The first difference information and the first feature output by the previous forward propagation branch are input into a first feature alignment module. The first feature alignment module distorts the first feature output by the previous forward propagation branch so as to align the first feature output by the previous forward propagation branch with the first difference information, and outputs the first alignment information.

[0133] The current frame and the first alignment information are input into a first residual module, and the first residual module outputs a first feature.

[0134] Optionally, the backward propagation branch includes a second optical flow estimation module, a second feature alignment module and a second residual module.

[0135] The backward propagation branch is specifically configured as follows: the current frame and the next frame including the backward propagation branch are input into the second optical flow estimation module, the second optical flow estimation module compares the feature difference between the current frame and the next frame including the backward propagation branch, and outputs second difference information.

[0136] The second difference information and the second feature output by the next backward propagation branch are input into the second feature alignment module. The second feature alignment module distorts the second feature output by the next backward propagation branch to align the next second feature with the second difference information, and outputs the second alignment information.

[0137] The current frame and the second alignment information are input into a second residual module, and the second residual module outputs a second feature.

[0138] Optionally, the key frame is super-resolved based on the key frame feature, the first feature, and the second feature corresponding to the key frame, specifically including:

[0139] The key frame features, the first features and the second features are rearranged, and super-resolution of the key frame is performed based on the sub-pixel convolution method.

[0140] Optionally, performing super-resolution on the key frame based on the sub-pixel convolution method further includes:

[0141] Based on the Charbonnier loss function, the difference between each point in the key frame HD image and the key frame is calculated as the loss value.

[0142] Optionally, after decoding the video file and determining the key frames and non-key frames, the method further includes:

[0143] The key frames are segmented to obtain segmented images.

[0144] The segmented images close to the first edge of the key frame are stitched to the second edge of the key frame.

[0145] The first edge and the second edge can be spliced ​​into a continuous image.

[0146] The present application also provides an immersive video super-resolution device, comprising:

[0147] The decoding module 601 is used to decode the video file and determine key frames and non-key frames.

[0148] The propagation branch allocation module 602 is used to allocate key frame propagation branches, forward propagation branches and backward propagation branches to key frames. The key frame propagation branch is configured to calculate the key frame features of the current key frame based on the previous key frame, the key frame features of the previous key frame and the current key frame. The forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch and the current frame. The backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch and the current frame.

[0149] A forward propagation branch and a backward propagation branch are allocated to the non-key frames, and only one of the forward propagation branch and the backward propagation branch is allocated to each non-key frame, and the propagation branches allocated to two adjacent non-key frames are different.

[0150] The key frame super-resolution module 603 is used to perform super-resolution on the key frame based on the key frame feature, the first feature and the second feature corresponding to the key frame to obtain a high-definition key frame.

[0151] The first super-resolution module 604 is used to perform super-resolution on the non-key frame based on the first feature corresponding to the non-key frame to obtain a first high-definition non-key frame.

[0152] The second super-resolution module 605 is used to perform super-resolution on the non-key frame based on the second feature corresponding to the non-key frame to obtain a second high-definition non-key frame.

[0153] The video generation module 606 is used to obtain a high-definition immersive video file based on the high-definition key frame, the first high-definition non-key frame and the second high-definition non-key frame.

[0154] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the above-described methods when executing the program.

[0155] The present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above methods.

[0156] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0157] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0158] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0159] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. An immersive video super-resolution method, comprising: Decode the video file and determine the key frames and non-key frames; A key frame propagation branch, a forward propagation branch and a backward propagation branch are allocated to the key frame; wherein the key frame propagation branch is configured to calculate the key frame feature of the current key frame based on the previous key frame, the key frame feature of the previous key frame and the current key frame; the forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch and the current frame; the backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch and the current frame; Allocating the forward propagation branch and the backward propagation branch to the non-key frames, and only allocating one of the forward propagation branch and the backward propagation branch to each non-key frame, and the propagation branches allocated to two adjacent non-key frames are different; Based on the key frame feature corresponding to the key frame, the first feature and the second feature, super-resolution is performed on the key frame to obtain a high-definition key frame; Based on the first feature corresponding to the non-key frame, super-resolution is performed on the non-key frame to obtain a first high-definition non-key frame; Based on the second feature corresponding to the non-key frame, super-resolution is performed on the non-key frame to obtain a second high-definition non-key frame; A high-definition immersive video file is obtained based on the high-definition key frame, the first high-definition non-key frame, and the second high-definition non-key frame.

2. The immersive video super-resolution method according to claim 1, wherein: The key frame propagation branch includes a key frame optical flow estimation module, a key frame feature alignment module and a key frame residual module; The key frame propagation branch is specifically configured to: input the key frame and the previous key frame into the key frame optical flow estimation module, and the key frame optical flow estimation module compares the feature difference between the key frame and the previous key frame, and outputs key frame difference information; Inputting the key frame difference information and the key frame features of the previous key frame into the key frame feature alignment module, the key frame feature alignment module distorts the previous key frame features to align the previous key frame features with the key frame difference information, and outputs key frame alignment information; The key frame and the key frame alignment information are input into the key frame residual module, and the key frame residual module outputs the key frame feature.

3. The immersive video super-resolution method according to claim 1, wherein: The forward propagation branch includes a first optical flow estimation module, a first feature alignment module and a first residual module; The forward propagation branch is specifically configured to: input the current frame and the previous frame including the forward propagation branch into the first optical flow estimation module, and the first optical flow estimation module compares the feature difference between the current frame and the previous frame including the forward propagation branch, and outputs first difference information; Inputting the first difference information and the first feature output by the previous forward propagation branch into the first feature alignment module, the first feature alignment module distorts the first feature output by the previous forward propagation branch to align the first feature output by the previous forward propagation branch with the first difference information, and outputs first alignment information; The current frame and the first alignment information are input into the first residual module, and the first residual module outputs the first feature.

4. The immersive video super-resolution method according to claim 1, wherein: The back propagation branch includes a second optical flow estimation module, a second feature alignment module and a second residual module; The backward propagation branch is specifically configured to: input the current frame and the next frame including the backward propagation branch into the second optical flow estimation module, and the second optical flow estimation module compares the feature difference between the current frame and the next frame including the backward propagation branch, and outputs second difference information; Inputting the second difference information and the second feature output by the next backward propagation branch into the second feature alignment module, the second feature alignment module distorts the second feature output by the next backward propagation branch to align the next second feature with the second difference information, and outputs second alignment information; The current frame and the second alignment information are input into the second residual module, and the second residual module outputs the second feature.

5. The immersive video super-resolution method according to claim 1, wherein: The performing super-resolution on the key frame based on the key frame feature corresponding to the key frame, the first feature, and the second feature specifically includes: The key frame features, the first features, and the second features are rearranged, and super-resolution is performed on the key frame based on a sub-pixel convolution method.

6. The immersive video super-resolution method according to claim 5, wherein: The super-resolution of the key frame based on the sub-pixel convolution method further includes: The difference between each point in the key frame high-definition image and the key frame is calculated based on the Charbonnier loss function as a loss value.

7. The immersive video super-resolution method according to claim 1, wherein: After decoding the video file and determining the key frames and non-key frames, it also includes: Segmenting the key frame to obtain segmented images; splicing the segmented image close to the first edge of the key frame to the second edge of the key frame; The first edge and the second edge can be spliced ​​into a continuous image.

8. An immersive video super-resolution device, comprising: A decoding module, used for decoding video files and determining key frames and non-key frames; A propagation branch allocation module, configured to allocate a key frame propagation branch, a forward propagation branch and a backward propagation branch to the key frame; wherein the key frame propagation branch is configured to calculate the key frame feature of the current key frame based on the previous key frame, the key frame feature of the previous key frame and the current key frame; the forward propagation branch is configured to calculate the first feature of the current frame based on the previous frame including the forward propagation branch, the first feature output by the previous forward propagation branch and the current frame; the backward propagation branch is configured to calculate the second feature of the current frame based on the next frame including the backward propagation branch, the second feature output by the next backward propagation branch and the current frame; and allocating the forward propagation branch and the backward propagation branch to the non-key frames, and only allocating one of the forward propagation branch and the backward propagation branch to each non-key frame, and the propagation branches allocated to two adjacent non-key frames are different; A key frame super-resolution module, configured to perform super-resolution on the key frame based on the key frame feature corresponding to the key frame, the first feature, and the second feature to obtain a high-definition key frame; A first super-resolution module, configured to perform super-resolution on the non-key frame based on the first feature corresponding to the non-key frame to obtain a first high-definition non-key frame; A second super-resolution module, configured to perform super-resolution on the non-key frame based on the second feature corresponding to the non-key frame to obtain a second high-definition non-key frame; A video generation module is used to obtain a high-definition immersive video file based on the high-definition key frame, the first high-definition non-key frame and the second high-definition non-key frame.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

Images