A VR Intelligent Data Caching Method and System Based on Viewport Predictability

Abstract

This invention provides a VR intelligent data caching method and system based on viewport predictability, including step S1: setting up a distributed intelligent storage management caching architecture based on the VR environment; the intelligent storage management caching architecture divides edge nodes into edge servers and data consumers according to function; step S2: predicting the user's future viewpoint based on the viewpoint decay mechanism and user parameter information; the viewpoint decay mechanism generates the user's FOV; the user parameter information includes the user's historical head trajectory information; step S3: caching popular user tiles based on a dynamic tiling caching method. This invention can perform efficient data management and storage on edge servers, while providing high-response support for data-intensive VR applications, significantly improving the accuracy and reliability of viewpoint prediction and tiling caching algorithms, and saving response time and bandwidth required when users request data.

Description

Technical Field

[0001] This invention relates to the field of VR data transmission, and more specifically, to a VR intelligent data caching method and system based on viewport predictability. Background Technology

[0002] With the rise of the concept of the Metaverse, 360° video and virtual reality (VR), as its infrastructure technologies, have once again received widespread attention in various fields such as entertainment, education, manufacturing, transportation, and healthcare.

[0003] VR, with its ability to provide users with immersive experiences, has demonstrated enormous commercial and industrial potential. Users typically watch VR videos through head-mounted displays for a better viewing experience. However, VR streaming currently faces two major challenges: ultra-high bandwidth and ultra-low latency; furthermore, VR head-mounted displays lack portability and mobility.

[0004] Currently, the two main technologies addressing the two major challenges of VR streaming are tile-based VR video streaming encoding and cloud / edge-based architectures for 360° VR video streaming. Tiling-based VR video streaming encoding involves dividing a two-dimensional plane obtained through equal-size rectangular projection into several tiles; in this case, the VR video stream only needs to transmit these tiles instead of the entire image, significantly saving bandwidth. The cloud / edge-based architecture for 360° VR video streaming can perform computationally intensive tasks on cloud / edge servers and stream the VR video to the user, thus significantly alleviating the computational and storage pressure on the user's VR device during processing.

[0005] However, in the future metaverse environment, VR will bear the responsibility of bridging the physical and virtual worlds, inevitably leading to an explosive growth in data. Traditional tile-based VR video stream encoding and cloud / edge architectures cannot efficiently store and retrieve data, significantly increasing the cost for users requesting data. Furthermore, since most VR devices are resource-constrained, and cloud / edge architectures for 360° VR video stream transmission heavily rely on the computing and caching capabilities of cloud / edge servers, there is a risk of single points of failure.

[0006] Chinese patent document CN113920277A discloses a CloudVR data transmission method and system based on edge computing storage, including: mapping panoramic images using a pyramidal projection method, dividing each panoramic image into multiple video versions with different field of view angles for storage according to viewpoint projection; acquiring and dynamically predicting the three-dimensional motion data of the user terminal device and calculating the user's perspective information; selecting the video version with the corresponding field of view angle according to the user's perspective information, performing pyramid mapping on the video version to obtain stereoscopic image data, calculating the viewpoint position movement vector corresponding to the viewpoint switching, and transmitting the stereoscopic image data to the user terminal device. However, this patent document uses techniques such as pyramid mapping, while this invention uses techniques such as viewpoint attenuation mechanisms. Therefore, the technology and method used in this invention are fundamentally different from those in the patent document. Summary of the Invention

[0007] In view of the shortcomings of the prior art, the purpose of this invention is to provide a VR intelligent data caching method and system based on viewport predictability.

[0008] A VR intelligent data caching method based on viewport predictability provided by the present invention includes:

[0009] Step S1: Set up a distributed intelligent storage management cache architecture based on the VR environment;

[0010] The intelligent storage management cache architecture divides edge nodes into edge servers and data consumers based on their functions.

[0011] Step S2: Predict the user's future viewpoint based on the viewpoint decay mechanism and the user's parameter information;

[0012] The viewpoint attenuation mechanism generates the user's field of view (FOV).

[0013] The user's parameter information includes the user's historical head trajectory information;

[0014] Step S3: Cache popular user tiles based on the dynamic tile caching method.

[0015] Preferably, the edge server includes a Master module and a data container; the Master module has communication and management functions; and the data container completes the storage tasks assigned by the Master module.

[0016] Preferably, step S2 extracts features based on the user's historical viewport information of the VR video being viewed, generates an initial tag matrix, and records the user's historical viewing information.

[0017] Preferably, the viewpoint attenuation mechanism selects a viewpoint attenuation parameter, attenuates the tiled labels covered by the user's viewport in circles on the initial label matrix with the user as the center, and generates an attenuated label matrix; and predicts the user's future viewpoint based on deep learning to generate the user's FOV.

[0018] Preferably, the dynamic tiling caching method stores data in a cloud or local data storage container based on the popularity of the predicted user's FOV; the Master's decision-making process is viewed as a Markov chain, with the Master choosing whether to cache the data in a local data storage container according to policy π, transforming the process of finding the optimal Master into an optimization problem:

[0019] π * (s)=argmax a∈A (Q(s,a)),s∈S;

[0020] Here, Q(s,a) represents the expected reward obtained by performing action a in state s.

[0021] A VR intelligent data caching system based on viewport predictability, provided by the present invention, includes:

[0022] Intelligent storage management cache architecture module: Set up a distributed intelligent storage management cache architecture, and divide edge nodes into edge servers and data consumers according to their functions;

[0023] Viewpoint prediction module: Predicts the user's future viewpoint based on the viewpoint decay mechanism and the user's parameter information;

[0024] The viewpoint attenuation mechanism generates the user's field of view (FOV).

[0025] The user's parameter information includes the user's historical head trajectory information;

[0026] Tiling caching module: Caches popular user tiles based on dynamic tiling caching method;

[0027] Communication management module: manages the request flow and data flow between the cloud and data consumers, as well as the data interaction with the adjacent edge server;

[0028] Multi-user data writing and mapping module: manages data storage containers and storage space, and supports multiple users writing to a single file.

[0029] Preferably, the edge server includes a Master module and a data container; the Master module has communication and management functions; and the data container completes the storage tasks assigned by the Master module.

[0030] Preferably, the viewpoint prediction module extracts features based on the user's historical viewport information of the VR video, generates an initial tag matrix, and records the user's historical viewing information.

[0031] Preferably, the viewpoint attenuation mechanism selects a viewpoint attenuation parameter, attenuates the tiled labels covered by the user's viewport in circles on the initial label matrix with the user as the center, and generates an attenuated label matrix; and predicts the user's future viewpoint based on deep learning to generate the user's FOV.

[0032] Preferably, the dynamic tiling caching method stores data in a cloud or local data storage container based on the popularity of the predicted user's FOV; the Master's decision-making process is viewed as a Markov chain, with the Master choosing whether to cache the data in a local data storage container according to policy π, transforming the process of finding the optimal Master into an optimization problem:

[0033] π * (s)=argmax a∈A (Q(s,a)),s∈S;

[0034] Here, Q(s,a) represents the expected reward obtained by performing action a in state s.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The method provided by this invention designs a viewpoint attenuation mechanism to convey the user's video viewing history information. The optimized caching structure can significantly improve the accuracy and reliability of viewpoint prediction and tiling caching algorithms for datasets with different head movement rates.

[0037] 2. The method provided by this invention proposes a distributed intelligent storage management cache architecture suitable for VR environments. The architecture utilizes the concept of distributed storage in cloud computing and applies it to edge computing to solve the problems of ultra-high bandwidth and ultra-low latency encountered in current VR transmission streams. By using distributed storage, efficient data management and storage can be performed on edge servers, while providing high-response support for data-intensive VR applications.

[0038] 3. The method provided by this invention is based on deep learning and reinforcement learning. It samples and labels the user's historical viewing information and further predicts the user's future viewpoint. At the same time, based on the popularity of VR viewed by the user, popular content is cached in a cache data container in a distributed intelligent storage management cache architecture, which significantly saves the response time and bandwidth required when the user requests data.

[0039] 4. The method and system provided by this invention can help solve the challenges of ultra-high bandwidth and ultra-low latency currently faced by VR transmission, and can be significantly effective even on datasets with different user head movement rates; the distributed intelligent storage management system provided in the VR environment offers a solution for realizing VR transmission under limited resource conditions at the edge; this invention establishes a more powerful and predictable intelligent tiling caching method in edge environments with limited resources, which is of great significance to the infrastructure construction of the metaverse in terms of VR transmission streams.

[0040] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0041] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0042] Figure 1 This is a schematic diagram of a distributed intelligent storage management cache architecture applicable to a VR environment in an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the viewpoint prediction and tiling caching algorithm based on deep learning and reinforcement learning in an embodiment of the present invention. Detailed Implementation

[0044] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0045] This invention discloses a VR intelligent data caching method based on viewport predictability, comprising:

[0046] Reference Figure 1 As shown, firstly, a distributed intelligent storage management cache architecture suitable for VR environment is designed, providing a system architecture design for predictable intelligent tiling caching methods.

[0047] The intelligent storage management cache architecture divides edge nodes into edge servers and data consumers based on their functions, while VR head-mounted displays are considered data consumers because they receive user requests and forward them to the edge server.

[0048] The edge server comprises a Master module with communication and management functions, and data containers that perform storage tasks assigned by the Master module. The Master module primarily has three functions: an AI-based intelligent viewpoint prediction algorithm and dynamic tiling cache, communication management, and multi-user data mapping and storage. The Master's most important function is to predict future viewpoints and dynamically store content based on customer requirements. By tracking changes in past user requests, the Master predicts future requests and dynamically saves the most popular content to reduce the bandwidth and latency required for 360° video streaming. Furthermore, the Master is responsible for managing request streams and data exchange with adjacent edge servers. The data mapping table facilitates data location, allows modification and deletion of stored data, and a single file can have multiple writers.

[0049] Reference Figure 2 As shown, based on the viewpoint decay mechanism and the user's historical head trajectory information, the user's future viewpoint is predicted. The output is a probability matrix of future viewpoints falling on a 2D tiled plane. The two tile centers with the highest probabilities are selected as the predicted future viewpoints, and a field of view (FOV) is generated and delivered to the user in advance. The accuracy of viewpoint prediction and the efficiency of pre-transmitted FOV can be balanced by adjusting the radius of the generated FOV.

[0050] The viewpoint attenuation mechanism first selects a viewpoint attenuation parameter and attenuates the tiled labels covered by the user's viewport in a loop around the initial label matrix, generating an attenuated label matrix. Then, based on deep learning, it predicts the user's future viewpoint to generate the user's FOV. Features are extracted from the user's historical viewport information in the VR video to generate an initial label matrix to record the user's historical viewing information. The 360° video is projected onto a rectangular plane to obtain a 360°×180° 2D plane, which is then divided into 162 equally sized 20°×20° tiles. This invention designs a 162×20 two-dimensional initial label matrix. For example, this method first samples the 360° video at a frequency of 10Hz, and uses a 2-second prediction window. The method predicts the user's future viewpoint based on changes in the user's viewpoint within this prediction window. In this method, the first dimension of the two-dimensional initial label matrix represents the 162 tiles that divide the 2D plane, and the second dimension represents the sampling in the prediction window. The initial label matrix is ​​a Boolean matrix, with tiles covered by the user's FOV marked as 1 and the rest marked as 0.

[0051] For example, the viewpoint attenuation mechanism processes the initial label matrix to obtain an optimized label matrix and pre-sets a viewpoint attenuation parameter δ. In the optimized label matrix, the tile containing the user's viewpoint is still marked as 1. For the other tiles covered by the FOV, the label is multiplied by δ for every 20° distance between the center of the tile and the viewpoint; that is, a distance of k*20° is labeled as δ. k The optimized label matrix serves as the input for the intelligent viewpoint prediction method.

[0052] Finally, using the previously obtained FOV as input, the popularity characteristics of each tile are statistically analyzed. Popular tiles are cached based on the dynamic tile caching method, so that when users request cached video content again in subsequent requests, they can directly retrieve it from the cache without retrieving it from the cloud, thereby reducing the latency required for VR transmission streams.

[0053] The dynamic tiling caching method stores data in the cloud or local data storage container based on the popularity of the predicted user's FOV. The Master's decision-making process is viewed as a Markov chain; the Master chooses whether to cache the data in the local data storage container according to policy π. This transforms the process of finding the optimal Master into an optimization problem.

[0054] π * (s)=argmax a∈A (Q(s,a)),s∈S;

[0055] Here, Q(s,a) represents the expected reward obtained by performing action a in state s.

[0056] The above are basic embodiments of the present invention. The technical solution of the present invention will be further described below through a preferred embodiment.

[0057] Reference Figure 1 As shown, firstly, a distributed intelligent storage management cache architecture suitable for VR environment is designed, providing a system architecture design for predictable intelligent tiling caching methods.

[0058] The intelligent storage management cache architecture divides edge nodes into edge servers and data consumers based on their functions, while VR head-mounted displays are considered data consumers because they receive user requests and forward them to the edge server.

[0059] The edge server further includes a Master module with communication and management functions and data containers that complete the storage tasks assigned by the Master module. The Master module mainly has three functions: an intelligent view point prediction algorithm based on artificial intelligence and dynamic tiling caching, communication management, and multi-user data mapping and storage. The most important function of the Master is to predict the future viewport and dynamically store content according to the requirements of customers. By tracking the changes in past user requests, the Master predicts future requests and dynamically saves the most popular content to reduce the bandwidth and latency required for 360° video streaming. In addition, the Master is also responsible for managing the request flow and data exchange with adjacent edge servers. The data mapping table can conveniently locate data, and at the same time it allows the modification and deletion of stored data, and a file can have multiple writers.

[0060] Referring to Figure 2 As shown, according to the view point attenuation mechanism and the historical head trajectory information of the user, the future view point of the user is predicted. The output is a probability matrix of a future view point falling on the 2D plane tiling. Select the two tiling centers with the highest probability as the predicted future view point, and generate the FOV and deliver it to the user in advance. The accuracy of view point prediction and the efficiency of pre-transmitting the FOV can be balanced by adjusting the radius of the generated FOV.

[0061] Among them, the view point attenuation mechanism first selects a view point attenuation parameter, and attenuates the tiling labels covered by the user's viewport circle by circle on the initial label matrix centered on the user to generate an attenuated label matrix, and predicts the future view point of the user based on deep learning to generate the FOV of the user.

[0062] Specifically, for the initial label matrix V, oversampling or undersampling operations are performed on the data set, and the data set format is converted to 10Hz. The dimension of the initial label matrix V is 162×20, where 162 represents 162 tiles in the 2D plane, and 20 represents 2s of time; 20 FOV information of the 2D plane is sampled in the prediction window. In the time prediction window, when the user's FOV covers all or part of the kth tile in the tth time frame, the element v k,t takes the value of 1, otherwise it is 0.

[0063] For the optimized label matrix H obtained after being processed by the view point attenuation mechanism, H is also a 162×20 matrix, where 162 represents 162 tiles in the 2D plane, and 20 represents 20 FOV information of the 2D plane sampled in the time prediction window. The view point attenuation mechanism pre-sets a view point attenuation parameter δ. In the matrix H, when the distance D between the center point of a tile and the center point of the tile where the user's view point is located satisfies (m - 1)·l < D ≤ m·l, the tile represented by this element in the matrix H is marked as δ m, where l is the side length of each tile. Using matrix H as input to the LSTM algorithm, the input yields a probability matrix, where each element represents the probability of predicting the future viewpoint in different tiles.

[0064] Finally, popular user tiles are cached using a dynamic tiling caching method to reduce the latency required for VR streaming.

[0065] Specifically, each Master independently designs its own optimization caching and content replacement strategy. For one of the Masters, n ∈ N, where N represents the total number of Masters, its state at a given time is s, where s ∈ S, and S represents all possible states. Each state is obtained by extracting specific features from the user's historical requests, and the state space is S. n This can be obtained by counting the number of requests for each tile from the start to the current time. When a user receives a new request w tk At that time, w tk Representing the k-th tile of frame t, the Master may encounter the following two possible situations:

[0066] a) The tile requested by the user is cached;

[0067] b) The tile requested by the user was not cached.

[0068] In the first case a), the Master will keep the current cache unchanged;

[0069] In the second scenario (b), the Master will have a possible set of operations A of size C+1. tk Where C is the size of the cache space. A tk The set contains element {A tk0 A tk1 ,...,A tki ,...,A tkC}, where A tk0 This indicates that the content cached on the Master remains unchanged, while A tki This indicates that all tiles of the i-th video cached on the Master will be replaced by the requested tile. Therefore, the action space A at time t... t That is, A t =∏ k∈K A tk Viewing the Master's decision-making process as a Markov chain, where the Master chooses whether to flatten and cache requests in its local data storage container based on policy π, finding the optimal Master's decision-making process transforms into an optimization problem:

[0070] π * (s)=argmaxa∈A (Q(s,a)),s∈S.

[0071] Here, Q(s,a) represents the expected reward obtained by performing action a in state s. The optimal caching strategy for the Master is obtained by iterating over this optimization problem using the reinforcement learning DQN method until convergence.

[0072] For the evaluation matrix, the performance of the intelligent caching method is evaluated using three metrics: viewpoint prediction accuracy, cache hit rate, and cache utilization.

[0073] Viewpoint prediction accuracy is the ratio of the sum of the number of FOVs generated by the intelligent viewpoint prediction algorithm to the sum of the total number of user's actual viewpoints; cache hit rate is the ratio of the cache tiles hit by the dynamic tiling cache algorithm to the total number of tiles; cache utilization is the ratio of the cache tiles hit by the dynamic tiling cache algorithm to the cache capacity.

[0074] This embodiment proposes a distributed intelligent storage management caching architecture suitable for VR environments, based on deep learning and reinforcement learning, and features a predictable intelligent tiling caching method for system architecture design. Utilizing intelligent viewpoint prediction and dynamic tiling caching algorithms, it can predict the user's future viewpoint while generating the user's field of view, and cache popular video content based on the popularity of user VR video content requests, significantly reducing the bandwidth and latency burden of VR video transmission streams. In summary, this invention establishes a more powerful and predictable intelligent tiling caching method in edge environments with limited resources, which is of great significance to the infrastructure construction of the metaverse in VR transmission streams.

[0075] The present invention also provides a viewport-predictable VR intelligent data caching system, which can be implemented by executing the process steps of the viewport-predictable VR intelligent data caching method. That is, those skilled in the art can understand the viewport-predictable VR intelligent data caching method as a preferred embodiment of the viewport-predictable VR intelligent data caching system.

[0076] Specifically, a VR intelligent data caching system based on viewport predictability includes:

[0077] Intelligent storage management cache architecture module: Set up a distributed intelligent storage management cache architecture, and divide edge nodes into edge servers and data consumers according to their functions;

[0078] Viewpoint prediction module: Predicts the user's future viewpoint based on the viewpoint decay mechanism and the user's parameter information;

[0079] The viewpoint attenuation mechanism generates the user's field of view (FOV).

[0080] The user's parameter information includes the user's historical head trajectory information;

[0081] Tiling caching module: Caches popular user tiles based on dynamic tiling caching method;

[0082] Communication management module: manages the request flow and data flow between the cloud and data consumers, as well as the data interaction with the adjacent edge server;

[0083] Multi-user data writing and mapping module: manages data storage containers and storage space, and supports multiple users writing to a single file.

[0084] The edge server includes a Master module and a data container; the Master module has communication and management functions; the data container completes the storage tasks assigned by the Master module.

[0085] The viewpoint prediction module extracts features based on the user's historical viewport information of the VR video they watched, and generates an initial tag matrix to record the user's historical viewing information.

[0086] The viewpoint attenuation mechanism selects a viewpoint attenuation parameter and attenuates the tiled labels covered by the user's viewport in circles on the initial label matrix, centered on the user, to generate an attenuated label matrix; based on deep learning, it predicts the user's future viewpoint and generates the user's FOV.

[0087] The dynamic tiling caching method stores data in the cloud or local data storage container based on the popularity of the predicted user's FOV; the Master's decision-making process is viewed as a Markov chain, where the Master chooses whether to cache the data in the local data storage container according to policy π, transforming the process of finding the optimal Master into an optimization problem:

[0088] π * (s)=argmax a∈A (Q(s,a)),s∈S;

[0089] Here, Q(s,a) represents the expected reward obtained by performing action a in state s.

[0090] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A VR intelligent data caching method based on viewport predictability, characterized in that, include: Step S1: Set up a distributed intelligent storage management cache architecture based on the VR environment; The intelligent storage management cache architecture divides edge nodes into edge servers and data consumers based on their functions. Step S2: Predict the user's future viewpoint based on the viewpoint decay mechanism and the user's parameter information; The viewpoint attenuation mechanism generates the user's field of view (FOV). The user's parameter information includes the user's historical head trajectory information; Step S3: Cache popular user tiles based on the dynamic tiling caching method; The viewpoint attenuation mechanism selects a viewpoint attenuation parameter and attenuates the tiled labels covered by the user's viewport in circles on the initial label matrix, centered on the user, to generate an attenuated label matrix; based on deep learning, it predicts the user's future viewpoint and generates the user's FOV.

2. The VR intelligent data caching method based on viewport predictability according to claim 1, characterized in that, The edge server includes a Master module and a data container; the Master module has communication and management functions; the data container completes the storage tasks assigned by the Master module.

3. The VR intelligent data caching method based on viewport predictability according to claim 1, characterized in that, Step S2 extracts features based on the user's historical viewport information of the VR video, generates an initial tag matrix, and records the user's historical viewing information.

4. The VR intelligent data caching method based on viewport predictability according to claim 1, characterized in that, The dynamic tiling caching method stores data in the cloud or local data storage container based on the popularity of the predicted user's FOV; the Master's decision-making process is viewed as a Markov chain, where the Master chooses whether to cache the data in the local data storage container according to policy π, transforming the process of finding the optimal Master into an optimization problem: ； Here, Q(s,a) represents the expected reward obtained by performing action a in state s.

5. A VR intelligent data caching system based on viewport predictability, characterized in that, include: Intelligent storage management cache architecture module: Set up a distributed intelligent storage management cache architecture, and divide edge nodes into edge servers and data consumers according to their functions; Viewpoint prediction module: Predicts the user's future viewpoint based on the viewpoint decay mechanism and the user's parameter information; The viewpoint attenuation mechanism generates the user's field of view (FOV). The user's parameter information includes the user's historical head trajectory information; Tiling caching module: Caches popular user tiles based on dynamic tiling caching method; Communication management module: manages the request flow and data flow between the cloud and data consumers, as well as the data interaction with the adjacent edge server; Multi-user data writing and mapping module: manages data storage containers and storage space, and supports multiple users writing to a single file; The viewpoint attenuation mechanism selects a viewpoint attenuation parameter and attenuates the tiled labels covered by the user's viewport in circles on the initial label matrix, centered on the user, to generate an attenuated label matrix; based on deep learning, it predicts the user's future viewpoint and generates the user's FOV.

6. A VR intelligent data caching system based on viewport predictability according to claim 5, characterized in that, The edge server includes a Master module and a data container; the Master module has communication and management functions; the data container completes the storage tasks assigned by the Master module.

7. A VR intelligent data caching system based on viewport predictability according to claim 5, characterized in that, The viewpoint prediction module extracts features based on the user's historical viewport information of the VR video they watched, generates an initial tag matrix, and records the user's historical viewing information.

8. A VR intelligent data caching system based on viewport predictability according to claim 5, characterized in that, The dynamic tiling caching method stores data in the cloud or local data storage container based on the popularity of the predicted user's FOV; the Master's decision-making process is viewed as a Markov chain, where the Master chooses whether to cache the data in the local data storage container according to policy π, transforming the process of finding the optimal Master into an optimization problem: ； Here, Q(s,a) represents the expected reward obtained by performing action a in state s.

Images